Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1445827 Details for
Bug 1581309
'test failed unknown error' when adding OSP Glance provider using https
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
/var/log/ovirt-engine/engine.log
engine.log (text/plain), 9.76 MB, created by
Dave
on 2018-05-30 12:28:04 UTC
(
hide
)
Description:
/var/log/ovirt-engine/engine.log
Filename:
MIME Type:
Creator:
Dave
Created:
2018-05-30 12:28:04 UTC
Size:
9.76 MB
patch
obsolete
>2018-05-29 03:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1a82e3f5] Autorecovering 2 hosts >2018-05-29 03:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1a82e3f5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [32ad692] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fff6fe5 >2018-05-29 03:10:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [32ad692] FINISH, SetVdsStatusVDSCommand, log id: fff6fe5 >2018-05-29 03:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [32ad692] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:10:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [32ad692] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:10:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [25ecb086] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1383380f >2018-05-29 03:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [25ecb086] FINISH, SetVdsStatusVDSCommand, log id: 1383380f >2018-05-29 03:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [25ecb086] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2b53e289 >2018-05-29 03:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [25ecb086] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2b53e289 >2018-05-29 03:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:10:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [25ecb086] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:10:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5cf7c216] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 592dc26d >2018-05-29 03:10:02,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5cf7c216] FINISH, GetHardwareInfoVDSCommand, log id: 592dc26d >2018-05-29 03:10:02,129-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7d7aee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 34e61635 >2018-05-29 03:10:02,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7d7aee] FINISH, GetHardwareInfoVDSCommand, log id: 34e61635 >2018-05-29 03:10:02,132-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [12b03804] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:10:02,142-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [71cd273b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:10:02,146-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [39abab60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:10:02,147-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1649b54b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:10:02,151-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [8780df1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:10:02,156-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [2a4ddb1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:10:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1649b54b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6892272c >2018-05-29 03:10:02,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [2a4ddb1d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 603ec9ee >2018-05-29 03:10:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:10:02,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [2a4ddb1d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 603ec9ee >2018-05-29 03:10:02,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:02,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:02,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:02,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:02,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:02,598-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [2a4ddb1d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:02,601-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [2a4ddb1d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:10:02,604-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a4ddb1d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a4ddb1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:10:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73e9bac0 >2018-05-29 03:10:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:10:02,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:10:02,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:10:02,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73e9bac0 >2018-05-29 03:10:02,708-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:10:02,721-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [532215ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:10:02,722-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [532215ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e9028af >2018-05-29 03:10:02,724-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [532215ab] FINISH, SetVdsStatusVDSCommand, log id: 1e9028af >2018-05-29 03:10:02,732-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 532215ab, Job ID: 68b38be4-5559-46d4-a3ce-9f2a20a3f770, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:10:02,737-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8780df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:10:02,739-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [532215ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8780df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:10:02,750-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [52244775] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:10:02,751-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [52244775] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:10:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:10:03,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1649b54b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6892272c >2018-05-29 03:10:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:03,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:03,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:03,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:10:03,503-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1649b54b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:10:03,506-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1649b54b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:10:03,509-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1649b54b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1649b54b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:10:03,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2063e584 >2018-05-29 03:10:03,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:10:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:10:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:10:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2063e584 >2018-05-29 03:10:03,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:10:03,623-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [52c2d1fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:10:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52c2d1fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eedb648 >2018-05-29 03:10:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52c2d1fa] FINISH, SetVdsStatusVDSCommand, log id: eedb648 >2018-05-29 03:10:03,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52c2d1fa, Job ID: 598771d5-a239-409e-a12f-169ce7e9a37e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:10:03,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71cd273b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:10:03,638-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52c2d1fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71cd273b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:10:03,649-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3d9612e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:10:03,650-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3d9612e1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [19c413c] Autorecovering 2 hosts >2018-05-29 03:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [19c413c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:15:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7c2fb44f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 745b3af9 >2018-05-29 03:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7c2fb44f] FINISH, SetVdsStatusVDSCommand, log id: 745b3af9 >2018-05-29 03:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7c2fb44f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:15:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [7c2fb44f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:15:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:15:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [35706c0e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f50fc51 >2018-05-29 03:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [35706c0e] FINISH, SetVdsStatusVDSCommand, log id: f50fc51 >2018-05-29 03:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [35706c0e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4820c29 >2018-05-29 03:15:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [35706c0e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4820c29 >2018-05-29 03:15:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:15:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [35706c0e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:15:00,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3280d2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d1700f8 >2018-05-29 03:15:00,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3280d2e4] FINISH, GetHardwareInfoVDSCommand, log id: 4d1700f8 >2018-05-29 03:15:00,413-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6afb5d72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:15:00,422-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3a9a5cd0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:15:00,426-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68dd6d79] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:15:00,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68dd6d79] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e15e54d >2018-05-29 03:15:00,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:15:00,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68dd6d79] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e15e54d >2018-05-29 03:15:00,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:00,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:00,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68dd6d79] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:00,832-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68dd6d79] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:15:00,834-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68dd6d79] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68dd6d79, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:15:00,841-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61bd51c9 >2018-05-29 03:15:00,841-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:15:00,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:15:00,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:15:00,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61bd51c9 >2018-05-29 03:15:00,937-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:15:00,949-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [59b863b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:15:00,950-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [59b863b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16dcbd21 >2018-05-29 03:15:00,953-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [59b863b6] FINISH, SetVdsStatusVDSCommand, log id: 16dcbd21 >2018-05-29 03:15:00,985-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59b863b6, Job ID: acecd04a-e3ba-423f-88ee-3ffbe5dde0bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:15:00,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3a9a5cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:15:00,992-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [59b863b6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3a9a5cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:15:01,003-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6e914885] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:15:01,004-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6e914885] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:15:01,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [293b14b1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4be83a8a >2018-05-29 03:15:01,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [293b14b1] FINISH, GetHardwareInfoVDSCommand, log id: 4be83a8a >2018-05-29 03:15:01,348-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [107779bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:15:01,357-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1fe383a1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:15:01,363-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3c3402da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:15:01,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3c3402da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7435d719 >2018-05-29 03:15:01,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:15:01,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3c3402da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7435d719 >2018-05-29 03:15:01,758-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:01,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:01,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:01,763-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:15:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3c3402da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:15:01,771-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3c3402da] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:15:01,774-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c3402da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c3402da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:15:01,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29396ef1 >2018-05-29 03:15:01,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:15:01,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:15:01,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:15:01,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29396ef1 >2018-05-29 03:15:01,871-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:15:01,884-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3c098215] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:15:01,885-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c098215] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e564a15 >2018-05-29 03:15:01,887-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c098215] FINISH, SetVdsStatusVDSCommand, log id: 4e564a15 >2018-05-29 03:15:01,893-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c098215, Job ID: 87b1c92e-6918-4378-bc25-dfdf4747365c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:15:01,897-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fe383a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:15:01,899-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c098215] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fe383a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:15:01,909-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2e7367c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:15:01,911-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2e7367c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1430a203] Autorecovering 2 hosts >2018-05-29 03:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1430a203] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [734f5e84] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25699886 >2018-05-29 03:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [734f5e84] FINISH, SetVdsStatusVDSCommand, log id: 25699886 >2018-05-29 03:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [734f5e84] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:20:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [734f5e84] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61fae362] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a5c53c1 >2018-05-29 03:20:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61fae362] FINISH, SetVdsStatusVDSCommand, log id: 4a5c53c1 >2018-05-29 03:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [61fae362] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1293a913 >2018-05-29 03:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [61fae362] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1293a913 >2018-05-29 03:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61fae362] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:20:01,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [17bfd319] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44aeb9e0 >2018-05-29 03:20:01,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [17bfd319] FINISH, GetHardwareInfoVDSCommand, log id: 44aeb9e0 >2018-05-29 03:20:01,661-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [44e66103] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:20:01,666-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [420ed963] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:20:01,671-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [505fba9a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:20:01,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [505fba9a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a09a879 >2018-05-29 03:20:02,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:20:02,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [505fba9a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a09a879 >2018-05-29 03:20:02,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:02,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:02,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:02,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:02,065-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:02,066-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [505fba9a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:02,069-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [505fba9a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:20:02,072-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [505fba9a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 505fba9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:20:02,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 515ac58c >2018-05-29 03:20:02,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:20:02,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:20:02,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:20:02,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:20:02,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:20:02,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 515ac58c >2018-05-29 03:20:02,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:20:02,190-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2be48ce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:20:02,191-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2be48ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f56856c >2018-05-29 03:20:02,194-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2be48ce] FINISH, SetVdsStatusVDSCommand, log id: 5f56856c >2018-05-29 03:20:02,225-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2be48ce, Job ID: e4993c48-fbb5-41d4-bdd3-31140e9f51bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:20:02,230-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 420ed963, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:20:02,233-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2be48ce] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 420ed963, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:20:02,244-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5e54b9fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:20:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5e54b9fe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:20:02,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [68c413ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3545f278 >2018-05-29 03:20:02,579-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [68c413ab] FINISH, GetHardwareInfoVDSCommand, log id: 3545f278 >2018-05-29 03:20:02,591-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [707aea05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:20:02,596-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [63d97c29] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:20:02,601-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [76dd858f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:20:02,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [76dd858f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: fca1ebf >2018-05-29 03:20:02,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:20:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [76dd858f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: fca1ebf >2018-05-29 03:20:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:02,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:02,995-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:02,996-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:03,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:20:03,000-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [76dd858f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:20:03,004-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [76dd858f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:20:03,006-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [76dd858f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76dd858f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:20:03,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 89f9187 >2018-05-29 03:20:03,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:20:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:20:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:20:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 89f9187 >2018-05-29 03:20:03,105-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:20:03,118-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [15544d6f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:20:03,119-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [15544d6f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2cf0ab3b >2018-05-29 03:20:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [15544d6f] FINISH, SetVdsStatusVDSCommand, log id: 2cf0ab3b >2018-05-29 03:20:03,127-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15544d6f, Job ID: 62c86305-040e-4983-a0e7-ac400a2cdfe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:20:03,132-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63d97c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:20:03,134-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [15544d6f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63d97c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:20:03,144-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7792a274] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:20:03,145-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7792a274] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5d7c9b91] Autorecovering 2 hosts >2018-05-29 03:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5d7c9b91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:25:00,036-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [113e6344] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32580ed5 >2018-05-29 03:25:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [113e6344] FINISH, SetVdsStatusVDSCommand, log id: 32580ed5 >2018-05-29 03:25:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [113e6344] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:25:00,051-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [113e6344] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:25:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d425e68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d4549f >2018-05-29 03:25:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d425e68] FINISH, SetVdsStatusVDSCommand, log id: 4d4549f >2018-05-29 03:25:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4d425e68] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6a0257e6 >2018-05-29 03:25:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4d425e68] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6a0257e6 >2018-05-29 03:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:25:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4d425e68] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:25:00,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ac08dd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 595695bf >2018-05-29 03:25:00,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ac08dd] FINISH, GetHardwareInfoVDSCommand, log id: 595695bf >2018-05-29 03:25:00,727-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [90daa54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:25:00,735-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6a58e8d1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:25:00,741-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1ce8f94c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:25:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1ce8f94c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79c64c9 >2018-05-29 03:25:01,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:25:01,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1ce8f94c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79c64c9 >2018-05-29 03:25:01,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:01,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:01,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:01,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:01,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:01,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1ce8f94c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:01,150-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1ce8f94c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:25:01,153-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ce8f94c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ce8f94c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:25:01,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 43b6df3 >2018-05-29 03:25:01,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:25:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:25:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:25:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 43b6df3 >2018-05-29 03:25:01,257-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:25:01,271-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5ed20c64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:25:01,272-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ed20c64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a8b7783 >2018-05-29 03:25:01,274-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ed20c64] FINISH, SetVdsStatusVDSCommand, log id: 5a8b7783 >2018-05-29 03:25:01,306-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ed20c64, Job ID: 61352822-d8a9-4823-9351-ee7798c4c6f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:25:01,311-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a58e8d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:25:01,313-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5ed20c64] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a58e8d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:25:01,323-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [48c2aa82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:25:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [48c2aa82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:25:02,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2b09c245] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42b10bda >2018-05-29 03:25:02,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2b09c245] FINISH, GetHardwareInfoVDSCommand, log id: 42b10bda >2018-05-29 03:25:02,969-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4625a336] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:25:02,973-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [282a4647] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:25:02,978-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6463d999] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:25:02,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6463d999] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 357b462a >2018-05-29 03:25:03,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:25:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6463d999] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 357b462a >2018-05-29 03:25:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:03,369-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:03,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:25:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6463d999] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:25:03,377-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6463d999] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:25:03,379-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6463d999] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6463d999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:25:03,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: cb8f171 >2018-05-29 03:25:03,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:25:03,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:25:03,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:25:03,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: cb8f171 >2018-05-29 03:25:03,479-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:25:03,492-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [906012a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:25:03,493-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [906012a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c4a241 >2018-05-29 03:25:03,496-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [906012a] FINISH, SetVdsStatusVDSCommand, log id: 41c4a241 >2018-05-29 03:25:03,501-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 906012a, Job ID: 99b294be-481c-409b-844a-d3f607302c87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:25:03,505-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 282a4647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:25:03,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [906012a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 282a4647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:25:03,518-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7bc980c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:25:03,519-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7bc980c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a5a91] Autorecovering 2 hosts >2018-05-29 03:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a5a91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:30:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:30:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:30:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6bf710b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ead38e9 >2018-05-29 03:30:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6bf710b] FINISH, SetVdsStatusVDSCommand, log id: 5ead38e9 >2018-05-29 03:30:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6bf710b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:30:00,051-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6bf710b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:30:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 347fc78 >2018-05-29 03:30:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] FINISH, SetVdsStatusVDSCommand, log id: 347fc78 >2018-05-29 03:30:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 102bb8d8 >2018-05-29 03:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6c52b27a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 102bb8d8 >2018-05-29 03:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6c52b27a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:30:01,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [645a275e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1448299a >2018-05-29 03:30:01,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [645a275e] FINISH, GetHardwareInfoVDSCommand, log id: 1448299a >2018-05-29 03:30:01,276-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [60b9924d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:30:01,285-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [28beddc0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:30:01,290-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6f1b0859] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:30:01,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6f1b0859] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7b718665 >2018-05-29 03:30:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:30:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6f1b0859] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7b718665 >2018-05-29 03:30:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:01,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:01,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:01,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:01,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:01,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6f1b0859] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:01,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6f1b0859] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:30:01,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6f1b0859] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6f1b0859, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:30:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7682432e >2018-05-29 03:30:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:30:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:30:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:30:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7682432e >2018-05-29 03:30:01,799-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:30:01,812-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d4fd5f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:30:01,813-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d4fd5f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fa9a319 >2018-05-29 03:30:01,815-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d4fd5f] FINISH, SetVdsStatusVDSCommand, log id: 2fa9a319 >2018-05-29 03:30:01,845-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d4fd5f, Job ID: c19db18f-86a7-4866-9784-4f0105b280d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:30:01,850-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 28beddc0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:30:01,852-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d4fd5f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 28beddc0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:30:01,862-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3a357293] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:30:01,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3a357293] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:30:01,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [19bf3608] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b1e6ccc >2018-05-29 03:30:01,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [19bf3608] FINISH, GetHardwareInfoVDSCommand, log id: 1b1e6ccc >2018-05-29 03:30:01,955-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [f8ed03b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:30:01,960-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d6558ff] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:30:01,965-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7caf5f5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:30:01,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7caf5f5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53694b83 >2018-05-29 03:30:02,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:30:02,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7caf5f5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53694b83 >2018-05-29 03:30:02,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:02,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:02,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:02,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:02,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:30:02,362-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7caf5f5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:30:02,365-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7caf5f5f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:30:02,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7caf5f5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7caf5f5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:30:02,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c10fc01 >2018-05-29 03:30:02,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:30:02,466-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:30:02,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:30:02,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:30:02,467-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:30:02,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c10fc01 >2018-05-29 03:30:02,467-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:30:02,479-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [23264963] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:30:02,481-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [23264963] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1650244a >2018-05-29 03:30:02,483-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [23264963] FINISH, SetVdsStatusVDSCommand, log id: 1650244a >2018-05-29 03:30:02,488-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23264963, Job ID: 29f42b5e-03b5-41f8-8a7a-d8fa131931cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:30:02,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d6558ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:30:02,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [23264963] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d6558ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:30:02,505-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [76d2c60e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:30:02,506-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [76d2c60e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2cfea204] Autorecovering 2 hosts >2018-05-29 03:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2cfea204] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:35:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3b655e80] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42c7815d >2018-05-29 03:35:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3b655e80] FINISH, SetVdsStatusVDSCommand, log id: 42c7815d >2018-05-29 03:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3b655e80] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:35:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3b655e80] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:35:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f7c23b7 >2018-05-29 03:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] FINISH, SetVdsStatusVDSCommand, log id: f7c23b7 >2018-05-29 03:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 18b9c11c >2018-05-29 03:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6d2e49bf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 18b9c11c >2018-05-29 03:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:35:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6d2e49bf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:35:02,478-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3d111d2b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3cb09756 >2018-05-29 03:35:02,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3d111d2b] FINISH, GetHardwareInfoVDSCommand, log id: 3cb09756 >2018-05-29 03:35:02,494-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [ec92e42] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:35:02,498-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5fba9753] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:35:02,504-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65cdcb27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:35:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65cdcb27] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35d561e8 >2018-05-29 03:35:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:35:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65cdcb27] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35d561e8 >2018-05-29 03:35:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:02,891-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:02,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:02,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:02,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:02,899-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65cdcb27] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:02,902-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65cdcb27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:35:02,905-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65cdcb27] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65cdcb27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:35:02,911-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1849615b >2018-05-29 03:35:02,911-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:35:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:35:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:35:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:35:03,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:35:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1849615b >2018-05-29 03:35:03,012-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:35:03,024-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [18dd62db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:35:03,025-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18dd62db] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57ad74c5 >2018-05-29 03:35:03,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18dd62db] FINISH, SetVdsStatusVDSCommand, log id: 57ad74c5 >2018-05-29 03:35:03,035-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18dd62db, Job ID: 4e3c72db-5531-42e9-9430-952f8ea3fc27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:35:03,039-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5fba9753, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:35:03,041-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18dd62db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5fba9753, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:35:03,051-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [b4f5172] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:35:03,052-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [b4f5172] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:35:03,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [211fc20c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 515314e0 >2018-05-29 03:35:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [211fc20c] FINISH, GetHardwareInfoVDSCommand, log id: 515314e0 >2018-05-29 03:35:03,085-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3b6707ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:35:03,090-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [228ed09b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:35:03,095-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3f1110e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:35:03,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3f1110e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ade2201 >2018-05-29 03:35:03,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:35:03,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3f1110e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ade2201 >2018-05-29 03:35:03,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:03,480-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:03,484-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:03,485-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:35:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3f1110e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:35:03,493-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3f1110e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:35:03,496-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f1110e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3f1110e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:35:03,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ad5b6f9 >2018-05-29 03:35:03,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:35:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:35:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:35:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ad5b6f9 >2018-05-29 03:35:03,594-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:35:03,606-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4ef54c2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:35:03,608-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4ef54c2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ac34a82 >2018-05-29 03:35:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4ef54c2c] FINISH, SetVdsStatusVDSCommand, log id: 3ac34a82 >2018-05-29 03:35:03,615-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4ef54c2c, Job ID: d90d2aff-d7b2-494c-9d6c-c560f0a7d9bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:35:03,620-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 228ed09b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:35:03,622-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4ef54c2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 228ed09b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:35:03,632-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3c0f857b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:35:03,634-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3c0f857b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler6) [1ae7fcb9] Start deleteAgedOutAuditLogs >2018-05-29 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler7) [2c942208] Start deleteAgedOutCommandEntities >2018-05-29 03:35:35,002-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler7) [2c942208] Finished deleteAgedOutCommandEntities >2018-05-29 03:35:35,002-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler6) [1ae7fcb9] Finished deleteAgedOutAuditLogs >2018-05-29 03:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [61f1e891] Autorecovering 2 hosts >2018-05-29 03:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [61f1e891] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:40:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1affe50d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b7f99b7 >2018-05-29 03:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1affe50d] FINISH, SetVdsStatusVDSCommand, log id: 3b7f99b7 >2018-05-29 03:40:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1affe50d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:40:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1affe50d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:40:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:40:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:40:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f586250 >2018-05-29 03:40:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] FINISH, SetVdsStatusVDSCommand, log id: 3f586250 >2018-05-29 03:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5f1dc209 >2018-05-29 03:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c2ff5c4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5f1dc209 >2018-05-29 03:40:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:40:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c2ff5c4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:40:00,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [484c219c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 555a3ad6 >2018-05-29 03:40:00,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [484c219c] FINISH, GetHardwareInfoVDSCommand, log id: 555a3ad6 >2018-05-29 03:40:00,730-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2bcf3fd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:40:00,739-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [b882ec7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:40:00,744-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [e797d73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:40:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [e797d73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce1a2d0 >2018-05-29 03:40:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:40:01,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [e797d73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce1a2d0 >2018-05-29 03:40:01,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,179-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [e797d73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,187-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [e797d73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:40:01,190-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e797d73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e797d73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:40:01,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ac0d962 >2018-05-29 03:40:01,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:40:01,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [414cd06] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5cfa2b70 >2018-05-29 03:40:01,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [414cd06] FINISH, GetHardwareInfoVDSCommand, log id: 5cfa2b70 >2018-05-29 03:40:01,265-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3e1dcf3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:40:01,270-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [51eebd9e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:40:01,274-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6af876a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:40:01,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6af876a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54ac8e4d >2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:40:01,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:40:01,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:40:01,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: ac0d962 >2018-05-29 03:40:01,295-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:40:01,308-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1e16965c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:40:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e16965c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23587e3f >2018-05-29 03:40:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e16965c] FINISH, SetVdsStatusVDSCommand, log id: 23587e3f >2018-05-29 03:40:01,344-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e16965c, Job ID: 5fe4c767-9d75-4076-ad98-4b221fb730e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:40:01,349-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b882ec7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:40:01,351-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e16965c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b882ec7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:40:01,361-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [142d51f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:40:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [142d51f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:40:01,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:40:01,657-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6af876a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54ac8e4d >2018-05-29 03:40:01,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:40:01,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6af876a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:40:01,673-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6af876a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:40:01,676-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6af876a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6af876a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:40:01,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e2eb181 >2018-05-29 03:40:01,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:40:01,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:40:01,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:40:01,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2eb181 >2018-05-29 03:40:01,778-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:40:01,789-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7fdf06c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:40:01,790-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fdf06c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45098578 >2018-05-29 03:40:01,793-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fdf06c0] FINISH, SetVdsStatusVDSCommand, log id: 45098578 >2018-05-29 03:40:01,798-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fdf06c0, Job ID: 793e88fe-41cf-4269-8dd8-4d27741359d7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:40:01,802-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51eebd9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:40:01,804-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fdf06c0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51eebd9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:40:01,814-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5ac0cec2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:40:01,815-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5ac0cec2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [415bbb35] Autorecovering 2 hosts >2018-05-29 03:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [415bbb35] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5848a29f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3412b4db >2018-05-29 03:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5848a29f] FINISH, SetVdsStatusVDSCommand, log id: 3412b4db >2018-05-29 03:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5848a29f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5848a29f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bfeea94 >2018-05-29 03:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] FINISH, SetVdsStatusVDSCommand, log id: 3bfeea94 >2018-05-29 03:45:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 726a2f62 >2018-05-29 03:45:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1d1b9a2e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 726a2f62 >2018-05-29 03:45:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d1b9a2e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:45:02,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [482150c6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c0391c3 >2018-05-29 03:45:02,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [482150c6] FINISH, GetHardwareInfoVDSCommand, log id: 1c0391c3 >2018-05-29 03:45:02,108-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1791cefb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:45:02,117-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [743cb761] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:45:02,122-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41ecf5e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:45:02,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41ecf5e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10310eb5 >2018-05-29 03:45:02,386-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [77e41bd6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5488bd3e >2018-05-29 03:45:02,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [77e41bd6] FINISH, GetHardwareInfoVDSCommand, log id: 5488bd3e >2018-05-29 03:45:02,401-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1819ae3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:45:02,405-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [74a2918] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:45:02,410-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29bfd24f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:45:02,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29bfd24f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 210160cc >2018-05-29 03:45:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:45:02,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41ecf5e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10310eb5 >2018-05-29 03:45:02,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41ecf5e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,524-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41ecf5e0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:45:02,527-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ecf5e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41ecf5e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:45:02,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 39417a0b >2018-05-29 03:45:02,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:45:02,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:45:02,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:45:02,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 39417a0b >2018-05-29 03:45:02,631-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:45:02,646-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5e00da6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:45:02,647-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e00da6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 278b8e29 >2018-05-29 03:45:02,649-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e00da6c] FINISH, SetVdsStatusVDSCommand, log id: 278b8e29 >2018-05-29 03:45:02,684-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e00da6c, Job ID: d3243ab9-c89e-416c-a0c9-e879e2f70453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:45:02,688-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 743cb761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:45:02,691-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5e00da6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 743cb761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:45:02,701-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [23c9c4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:45:02,703-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [23c9c4b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:45:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:45:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29bfd24f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 210160cc >2018-05-29 03:45:02,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,802-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,806-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:45:02,811-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29bfd24f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:45:02,814-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29bfd24f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:45:02,817-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29bfd24f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29bfd24f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:45:02,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a219251 >2018-05-29 03:45:02,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:45:02,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:45:02,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:45:02,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a219251 >2018-05-29 03:45:02,919-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:45:02,932-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [56550ca2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:45:02,933-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [56550ca2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5202e5ab >2018-05-29 03:45:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [56550ca2] FINISH, SetVdsStatusVDSCommand, log id: 5202e5ab >2018-05-29 03:45:02,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56550ca2, Job ID: 411bbbfd-bfbd-4732-a6c4-79a40ff85dd2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:45:02,947-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74a2918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:45:02,949-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [56550ca2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74a2918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:45:02,960-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [67b0ab05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:45:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [67b0ab05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:50:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [121e3f7] Autorecovering 2 hosts >2018-05-29 03:50:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [121e3f7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:50:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:50:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:50:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [653f701c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76c7b545 >2018-05-29 03:50:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [653f701c] FINISH, SetVdsStatusVDSCommand, log id: 76c7b545 >2018-05-29 03:50:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [653f701c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:50:00,048-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [653f701c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:50:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:50:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:50:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cb54f42 >2018-05-29 03:50:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] FINISH, SetVdsStatusVDSCommand, log id: 7cb54f42 >2018-05-29 03:50:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6dd069fa >2018-05-29 03:50:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [46ebd0f8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6dd069fa >2018-05-29 03:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [46ebd0f8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:50:00,364-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b8f5905] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5a2978b2 >2018-05-29 03:50:00,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b8f5905] FINISH, GetHardwareInfoVDSCommand, log id: 5a2978b2 >2018-05-29 03:50:00,378-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [12408f96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:50:00,387-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5355cc17] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:50:00,392-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1c705773] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:50:00,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1c705773] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e8cc25c >2018-05-29 03:50:00,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [489399b2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 40e58b76 >2018-05-29 03:50:00,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [489399b2] FINISH, GetHardwareInfoVDSCommand, log id: 40e58b76 >2018-05-29 03:50:00,533-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [122e46a5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:50:00,537-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [137bb5c5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:50:00,543-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4675a3e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:50:00,549-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4675a3e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 435510d4 >2018-05-29 03:50:00,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:50:00,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4675a3e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 435510d4 >2018-05-29 03:50:00,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:00,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:00,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:00,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:00,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:00,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4675a3e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:00,946-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4675a3e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:50:00,949-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4675a3e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4675a3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:50:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d3056af >2018-05-29 03:50:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:50:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:50:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:50:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d3056af >2018-05-29 03:50:01,051-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:50:01,065-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [a3d86df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:50:01,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [a3d86df] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 679ae284 >2018-05-29 03:50:01,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [a3d86df] FINISH, SetVdsStatusVDSCommand, log id: 679ae284 >2018-05-29 03:50:01,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a3d86df, Job ID: 34e9620c-584c-4fd6-81b8-8d01d5a896a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:50:01,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 137bb5c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:50:01,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a3d86df] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 137bb5c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:50:01,116-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6430f565] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:50:01,117-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6430f565] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:50:02,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:50:02,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1c705773] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e8cc25c >2018-05-29 03:50:02,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:02,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:02,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:02,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:50:02,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1c705773] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:50:02,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1c705773] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:50:02,787-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c705773] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1c705773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:50:02,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ce8ece >2018-05-29 03:50:02,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:50:02,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:50:02,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:50:02,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: ce8ece >2018-05-29 03:50:02,890-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:50:02,903-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1d5d8dca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:50:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d5d8dca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34010f76 >2018-05-29 03:50:02,907-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d5d8dca] FINISH, SetVdsStatusVDSCommand, log id: 34010f76 >2018-05-29 03:50:02,912-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d5d8dca, Job ID: ffdaf84c-6330-4260-87cf-ed23f1e668d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:50:02,917-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5355cc17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:50:02,919-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d5d8dca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5355cc17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:50:02,930-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7b9d0be4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:50:02,931-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7b9d0be4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:54:42,811-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [43207b46] Attempting to update VMs/Templates Ovf. >2018-05-29 03:54:42,813-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 03:54:42,813-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 03:54:42,814-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:54:42,818-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 03:54:42,822-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 03:54:42,822-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 03:54:42,822-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 03:54:42,822-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 03:54:42,823-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 03:54:42,823-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [555de736] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 03:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6d26fb54] Autorecovering 2 hosts >2018-05-29 03:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6d26fb54] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:55:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [675c831] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ab778b1 >2018-05-29 03:55:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [675c831] FINISH, SetVdsStatusVDSCommand, log id: 4ab778b1 >2018-05-29 03:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [675c831] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 03:55:00,071-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [675c831] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 03:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47a2cf98 >2018-05-29 03:55:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] FINISH, SetVdsStatusVDSCommand, log id: 47a2cf98 >2018-05-29 03:55:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6020fcb1 >2018-05-29 03:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2724f5d2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6020fcb1 >2018-05-29 03:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 03:55:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2724f5d2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 03:55:00,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e13cbef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e311857 >2018-05-29 03:55:00,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e13cbef] FINISH, GetHardwareInfoVDSCommand, log id: 7e311857 >2018-05-29 03:55:00,600-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4b2ccd6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:55:00,608-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3b40e6b0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:55:00,613-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [78fb4dc5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:55:00,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [78fb4dc5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 418a6bc7 >2018-05-29 03:55:00,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:55:00,992-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [78fb4dc5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 418a6bc7 >2018-05-29 03:55:00,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:00,996-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:01,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:01,001-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:01,005-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [78fb4dc5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:01,009-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [78fb4dc5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:55:01,011-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fb4dc5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78fb4dc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:55:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31e90e18 >2018-05-29 03:55:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:55:01,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:55:01,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:55:01,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31e90e18 >2018-05-29 03:55:01,115-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:55:01,128-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7ba5047d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:55:01,130-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7ba5047d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 281ca77 >2018-05-29 03:55:01,132-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7ba5047d] FINISH, SetVdsStatusVDSCommand, log id: 281ca77 >2018-05-29 03:55:01,161-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ba5047d, Job ID: 1288c809-d60d-4e9c-8512-0d4058a25b48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:55:01,166-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b40e6b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:55:01,169-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7ba5047d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b40e6b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:55:01,180-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4c9f54a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 03:55:01,181-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4c9f54a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 03:55:01,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [e69b249] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 25b0c650 >2018-05-29 03:55:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [e69b249] FINISH, GetHardwareInfoVDSCommand, log id: 25b0c650 >2018-05-29 03:55:01,798-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [54d8385b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:55:01,808-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [57c512df] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:55:01,813-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [34046212] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 03:55:01,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [34046212] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 427ae7b0 >2018-05-29 03:55:02,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 03:55:02,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [34046212] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 427ae7b0 >2018-05-29 03:55:02,204-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:02,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:02,209-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:02,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 03:55:02,214-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [34046212] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 03:55:02,216-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [34046212] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 03:55:02,219-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34046212] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 34046212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 03:55:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44b59bff >2018-05-29 03:55:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:55:02,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 03:55:02,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 03:55:02,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44b59bff >2018-05-29 03:55:02,325-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 03:55:02,338-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [639e10b3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:55:02,339-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [639e10b3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54c1ba12 >2018-05-29 03:55:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [639e10b3] FINISH, SetVdsStatusVDSCommand, log id: 54c1ba12 >2018-05-29 03:55:02,373-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 639e10b3, Job ID: d3e779b4-f4ac-43bb-bb1b-b79c14d4be53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 03:55:02,378-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 57c512df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 03:55:02,380-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [639e10b3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 57c512df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 03:55:02,391-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [34e74b88] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 03:55:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [34e74b88] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [f881735] Autorecovering 2 hosts >2018-05-29 04:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [f881735] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:00:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:00:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [602f02a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e30c8ab >2018-05-29 04:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [602f02a4] FINISH, SetVdsStatusVDSCommand, log id: 2e30c8ab >2018-05-29 04:00:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [602f02a4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:00:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [602f02a4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:00:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:00:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:00:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [32f3b05] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d355584 >2018-05-29 04:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [32f3b05] FINISH, SetVdsStatusVDSCommand, log id: 4d355584 >2018-05-29 04:00:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [32f3b05] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c1cf926 >2018-05-29 04:00:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [32f3b05] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c1cf926 >2018-05-29 04:00:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [32f3b05] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:00:01,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4db7b2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d05189e >2018-05-29 04:00:01,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4db7b2e4] FINISH, GetHardwareInfoVDSCommand, log id: 4d05189e >2018-05-29 04:00:01,829-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [221eb62a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:00:01,838-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [50216119] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:00:01,843-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [245ad7de] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:00:01,855-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [245ad7de] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: a60e848 >2018-05-29 04:00:02,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b23382] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 225441b0 >2018-05-29 04:00:02,978-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [41b23382] FINISH, GetHardwareInfoVDSCommand, log id: 225441b0 >2018-05-29 04:00:02,991-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [50f1875d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:00:02,996-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [168480cc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:00:03,001-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [83d02b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:00:03,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [83d02b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 75eb3173 >2018-05-29 04:00:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:00:03,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [245ad7de] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: a60e848 >2018-05-29 04:00:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [245ad7de] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,247-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [245ad7de] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:00:03,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [245ad7de] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 245ad7de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:00:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22d295b9 >2018-05-29 04:00:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:00:03,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:00:03,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:00:03,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22d295b9 >2018-05-29 04:00:03,351-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:00:03,364-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [39ed2683] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:00:03,366-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [39ed2683] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ef56c93 >2018-05-29 04:00:03,368-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [39ed2683] FINISH, SetVdsStatusVDSCommand, log id: 4ef56c93 >2018-05-29 04:00:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:00:03,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [83d02b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 75eb3173 >2018-05-29 04:00:03,387-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,388-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:00:03,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [83d02b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:00:03,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39ed2683, Job ID: bda9ddcf-f491-4d0b-9804-1d4d9553a73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:00:03,400-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [83d02b0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:00:03,404-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50216119, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:00:03,405-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [83d02b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 83d02b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:00:03,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [39ed2683] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50216119, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:00:03,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2897b611 >2018-05-29 04:00:03,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:00:03,419-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [487c0bae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:00:03,420-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [487c0bae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:00:03,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:00:03,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:00:03,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2897b611 >2018-05-29 04:00:03,507-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:00:03,519-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [19442466] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:00:03,521-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [19442466] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fe5a1ea >2018-05-29 04:00:03,523-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [19442466] FINISH, SetVdsStatusVDSCommand, log id: 1fe5a1ea >2018-05-29 04:00:03,528-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 19442466, Job ID: 57e675c5-5737-4488-8284-71494c741110, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:00:03,533-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 168480cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:00:03,535-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [19442466] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 168480cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:00:03,545-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48f45aef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:00:03,546-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48f45aef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [16f87cdd] Autorecovering 2 hosts >2018-05-29 04:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [16f87cdd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:05:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [42c8f9b2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61c2aebc >2018-05-29 04:05:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [42c8f9b2] FINISH, SetVdsStatusVDSCommand, log id: 61c2aebc >2018-05-29 04:05:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [42c8f9b2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:05:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [42c8f9b2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13aeee66] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77914af1 >2018-05-29 04:05:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13aeee66] FINISH, SetVdsStatusVDSCommand, log id: 77914af1 >2018-05-29 04:05:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13aeee66] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5617c7b7 >2018-05-29 04:05:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13aeee66] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5617c7b7 >2018-05-29 04:05:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:05:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13aeee66] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:05:01,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3bc86d71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4a8a08ee >2018-05-29 04:05:01,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3bc86d71] FINISH, GetHardwareInfoVDSCommand, log id: 4a8a08ee >2018-05-29 04:05:01,125-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3bede82c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:05:01,133-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [7151e453] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:05:01,138-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21227971] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:05:01,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68791671] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 394769a7 >2018-05-29 04:05:01,145-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68791671] FINISH, GetHardwareInfoVDSCommand, log id: 394769a7 >2018-05-29 04:05:01,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21227971] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f65bc8c >2018-05-29 04:05:01,157-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4e2873f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:05:01,162-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [71fc162c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:05:01,167-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4a45a80e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:05:01,179-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4a45a80e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 120015de >2018-05-29 04:05:01,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:05:01,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21227971] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f65bc8c >2018-05-29 04:05:01,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,527-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,527-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,532-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21227971] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,534-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21227971] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:05:01,537-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21227971] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21227971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:05:01,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28f822c >2018-05-29 04:05:01,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:05:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:05:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4a45a80e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 120015de >2018-05-29 04:05:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:05:01,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4a45a80e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:05:01,580-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4a45a80e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:05:01,582-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a45a80e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4a45a80e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:05:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:05:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:05:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28f822c >2018-05-29 04:05:01,641-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:05:01,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f9b00a >2018-05-29 04:05:01,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:05:01,654-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6650a349] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:05:01,655-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6650a349] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3398d046 >2018-05-29 04:05:01,658-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6650a349] FINISH, SetVdsStatusVDSCommand, log id: 3398d046 >2018-05-29 04:05:01,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6650a349, Job ID: 76f748a9-7328-497b-8914-bc71bdc7902c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:05:01,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7151e453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:05:01,696-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6650a349] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7151e453, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:05:01,707-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3bc139b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:05:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3bc139b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:05:01,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:05:01,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:05:01,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f9b00a >2018-05-29 04:05:01,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:05:01,761-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [52e2cc9a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:05:01,762-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52e2cc9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52b4108f >2018-05-29 04:05:01,764-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [52e2cc9a] FINISH, SetVdsStatusVDSCommand, log id: 52b4108f >2018-05-29 04:05:01,771-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52e2cc9a, Job ID: b751dd75-6ae6-46ee-8552-5650f73dcbc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:05:01,775-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71fc162c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:05:01,777-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [52e2cc9a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71fc162c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:05:01,787-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3c6a20d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:05:01,788-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3c6a20d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1a417fe5] Autorecovering 2 hosts >2018-05-29 04:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1a417fe5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [19df4298] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a82ccd5 >2018-05-29 04:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [19df4298] FINISH, SetVdsStatusVDSCommand, log id: 6a82ccd5 >2018-05-29 04:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [19df4298] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:10:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [19df4298] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:10:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [225c88be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d697497 >2018-05-29 04:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [225c88be] FINISH, SetVdsStatusVDSCommand, log id: 3d697497 >2018-05-29 04:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [225c88be] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 99d5f16 >2018-05-29 04:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [225c88be] FINISH, SetHaMaintenanceModeVDSCommand, log id: 99d5f16 >2018-05-29 04:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [225c88be] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:10:02,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [554c4ec6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f741b1f >2018-05-29 04:10:02,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [554c4ec6] FINISH, GetHardwareInfoVDSCommand, log id: 1f741b1f >2018-05-29 04:10:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [18105fa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4312edea >2018-05-29 04:10:02,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [18105fa] FINISH, GetHardwareInfoVDSCommand, log id: 4312edea >2018-05-29 04:10:02,702-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5f47d028] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:10:02,706-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [37611b6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:10:02,711-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4a848800] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:10:02,715-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [697051e4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:10:02,717-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [24552673] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:10:02,721-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [408834cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:10:02,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [24552673] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53f119c3 >2018-05-29 04:10:02,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [408834cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce7df6d >2018-05-29 04:10:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:10:03,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [408834cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce7df6d >2018-05-29 04:10:03,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,107-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:10:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [24552673] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53f119c3 >2018-05-29 04:10:03,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,114-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,115-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,117-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [408834cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,120-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [408834cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:10:03,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [408834cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 408834cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:10:03,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:10:03,125-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [24552673] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:10:03,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5da98926 >2018-05-29 04:10:03,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:10:03,128-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [24552673] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:10:03,130-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24552673] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24552673, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:10:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:10:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:10:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5da98926 >2018-05-29 04:10:03,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:10:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5540c7cc >2018-05-29 04:10:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:10:03,238-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [710588c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:10:03,239-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [710588c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53972f97 >2018-05-29 04:10:03,241-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [710588c] FINISH, SetVdsStatusVDSCommand, log id: 53972f97 >2018-05-29 04:10:03,271-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 710588c, Job ID: 081febde-a2a1-4461-842b-32afa14d83bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:10:03,275-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 697051e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:10:03,278-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [710588c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 697051e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:10:03,288-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [36a9bfde] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:10:03,290-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [36a9bfde] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:10:03,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:10:03,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:10:03,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5540c7cc >2018-05-29 04:10:03,326-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:10:03,339-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [423b397c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:10:03,340-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [423b397c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61d22fff >2018-05-29 04:10:03,342-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [423b397c] FINISH, SetVdsStatusVDSCommand, log id: 61d22fff >2018-05-29 04:10:03,348-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 423b397c, Job ID: de0e5842-794b-488b-8676-34b304bc5e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:10:03,352-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a848800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:10:03,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [423b397c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a848800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:10:03,364-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1ef23871] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:10:03,365-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1ef23871] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [18a9448d] Autorecovering 2 hosts >2018-05-29 04:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [18a9448d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [426943a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a9afe45 >2018-05-29 04:15:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [426943a5] FINISH, SetVdsStatusVDSCommand, log id: 1a9afe45 >2018-05-29 04:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [426943a5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:15:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [426943a5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:15:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [14b083e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35647456 >2018-05-29 04:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [14b083e5] FINISH, SetVdsStatusVDSCommand, log id: 35647456 >2018-05-29 04:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [14b083e5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 24767e7f >2018-05-29 04:15:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [14b083e5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 24767e7f >2018-05-29 04:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:15:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [14b083e5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:15:00,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2938fbeb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e1b2efb >2018-05-29 04:15:00,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2938fbeb] FINISH, GetHardwareInfoVDSCommand, log id: 4e1b2efb >2018-05-29 04:15:00,914-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [71adde5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6370f444 >2018-05-29 04:15:00,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [71adde5d] FINISH, GetHardwareInfoVDSCommand, log id: 6370f444 >2018-05-29 04:15:00,930-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [619a2101] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:15:00,931-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2be741a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:15:00,936-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [51742a47] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:15:00,939-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7d50309b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:15:00,941-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [62982b89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:15:00,944-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1e25a5bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:15:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1e25a5bc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ffe5c59 >2018-05-29 04:15:01,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [62982b89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5cfce7e2 >2018-05-29 04:15:01,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:15:01,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1e25a5bc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ffe5c59 >2018-05-29 04:15:01,333-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,334-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,339-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,342-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,343-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1e25a5bc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,346-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1e25a5bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:15:01,348-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1e25a5bc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e25a5bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:15:01,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25bb7706 >2018-05-29 04:15:01,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:15:01,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:15:01,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [62982b89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5cfce7e2 >2018-05-29 04:15:01,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,387-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,391-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:15:01,396-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [62982b89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:15:01,399-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [62982b89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:15:01,401-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [62982b89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62982b89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:15:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:15:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:15:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25bb7706 >2018-05-29 04:15:01,457-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:15:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bc905e >2018-05-29 04:15:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:15:01,470-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6cd73f45] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:15:01,471-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd73f45] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45b67f98 >2018-05-29 04:15:01,473-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd73f45] FINISH, SetVdsStatusVDSCommand, log id: 45b67f98 >2018-05-29 04:15:01,504-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6cd73f45, Job ID: e285b416-6c9d-42c6-9d6c-0ab0f8f0be95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:15:01,509-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d50309b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:15:01,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6cd73f45] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d50309b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:15:01,522-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [535d69af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:15:01,523-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [535d69af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:15:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:15:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:15:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bc905e >2018-05-29 04:15:01,557-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:15:01,569-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7925efeb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:15:01,571-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7925efeb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 183167da >2018-05-29 04:15:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7925efeb] FINISH, SetVdsStatusVDSCommand, log id: 183167da >2018-05-29 04:15:01,578-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7925efeb, Job ID: bf6506f5-18a7-434f-a881-0d1e9cf7dd3f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:15:01,582-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51742a47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:15:01,585-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7925efeb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51742a47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:15:01,595-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [373fec35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:15:01,596-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [373fec35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7153284b] Autorecovering 2 hosts >2018-05-29 04:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7153284b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [358b767c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55fa9865 >2018-05-29 04:20:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [358b767c] FINISH, SetVdsStatusVDSCommand, log id: 55fa9865 >2018-05-29 04:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [358b767c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:20:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [358b767c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51e1739f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91bdac2 >2018-05-29 04:20:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51e1739f] FINISH, SetVdsStatusVDSCommand, log id: 91bdac2 >2018-05-29 04:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [51e1739f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 464dcca4 >2018-05-29 04:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [51e1739f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 464dcca4 >2018-05-29 04:20:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51e1739f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:20:02,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [35250b49] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24ee8251 >2018-05-29 04:20:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [23c1675c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ecaf582 >2018-05-29 04:20:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [35250b49] FINISH, GetHardwareInfoVDSCommand, log id: 24ee8251 >2018-05-29 04:20:02,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [23c1675c] FINISH, GetHardwareInfoVDSCommand, log id: 3ecaf582 >2018-05-29 04:20:02,211-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2f4da17b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:20:02,212-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1a5d91ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:20:02,221-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [35d7c9b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:20:02,221-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [fcbdfa2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:20:02,226-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ba0ad23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:20:02,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f3baffe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:20:02,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ba0ad23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a8f7e42 >2018-05-29 04:20:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f3baffe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 569f79b9 >2018-05-29 04:20:02,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:20:02,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f3baffe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 569f79b9 >2018-05-29 04:20:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:02,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:02,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:02,644-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:02,644-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f3baffe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:02,647-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f3baffe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:20:02,650-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f3baffe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f3baffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:20:02,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38d76dbe >2018-05-29 04:20:02,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:20:02,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:20:02,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:20:02,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38d76dbe >2018-05-29 04:20:02,755-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:20:02,769-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [15d0db41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:20:02,770-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d0db41] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d233c2e >2018-05-29 04:20:02,772-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d0db41] FINISH, SetVdsStatusVDSCommand, log id: 4d233c2e >2018-05-29 04:20:02,804-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15d0db41, Job ID: ec7f5858-2158-48ed-9003-3429aaad816a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:20:02,808-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: fcbdfa2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:20:02,810-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15d0db41] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: fcbdfa2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:20:02,821-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [606df59b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:20:02,822-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [606df59b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:20:03,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:20:03,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ba0ad23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a8f7e42 >2018-05-29 04:20:03,617-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:03,617-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:03,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:03,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:20:03,626-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ba0ad23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:20:03,630-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ba0ad23] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:20:03,633-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ba0ad23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ba0ad23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:20:03,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57278dfb >2018-05-29 04:20:03,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:20:03,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:20:03,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:20:03,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57278dfb >2018-05-29 04:20:03,734-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:20:03,748-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6dd16d97] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:20:03,749-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6dd16d97] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d7f05f0 >2018-05-29 04:20:03,751-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6dd16d97] FINISH, SetVdsStatusVDSCommand, log id: 5d7f05f0 >2018-05-29 04:20:03,756-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6dd16d97, Job ID: f2e4e083-b454-4d1c-ae50-b63e04de73ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:20:03,761-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35d7c9b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:20:03,763-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6dd16d97] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35d7c9b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:20:03,773-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [7f2fe61a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:20:03,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [7f2fe61a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [454bbedd] Autorecovering 2 hosts >2018-05-29 04:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [454bbedd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30b8c251] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 134d1805 >2018-05-29 04:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30b8c251] FINISH, SetVdsStatusVDSCommand, log id: 134d1805 >2018-05-29 04:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30b8c251] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:25:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [30b8c251] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:25:00,136-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:25:00,143-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:25:00,143-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,143-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,145-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65a88fd6 >2018-05-29 04:25:00,147-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] FINISH, SetVdsStatusVDSCommand, log id: 65a88fd6 >2018-05-29 04:25:00,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a76e03c >2018-05-29 04:25:00,152-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [78cc1ec9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a76e03c >2018-05-29 04:25:00,152-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:25:00,154-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78cc1ec9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:25:00,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [119f1299] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2c9dbbd5 >2018-05-29 04:25:00,444-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [119f1299] FINISH, GetHardwareInfoVDSCommand, log id: 2c9dbbd5 >2018-05-29 04:25:00,456-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [216c6169] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:25:00,465-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6d676a07] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:25:00,470-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [297cda2b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:25:00,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [297cda2b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c458299 >2018-05-29 04:25:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:25:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [297cda2b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c458299 >2018-05-29 04:25:00,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:00,859-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:00,863-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:00,864-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:00,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:00,868-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [297cda2b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:00,870-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [297cda2b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:25:00,874-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [297cda2b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 297cda2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:25:00,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55b0384f >2018-05-29 04:25:00,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:25:00,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:25:00,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:25:00,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55b0384f >2018-05-29 04:25:00,975-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:25:00,989-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5223efd7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:25:00,990-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5223efd7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e5e1b36 >2018-05-29 04:25:00,993-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5223efd7] FINISH, SetVdsStatusVDSCommand, log id: 2e5e1b36 >2018-05-29 04:25:01,021-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5223efd7, Job ID: 1057e5a7-68bc-4f0b-b0f9-fda624a346fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:25:01,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d676a07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:25:01,029-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5223efd7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d676a07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:25:01,040-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [17f95016] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:25:01,041-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [17f95016] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:25:01,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [bf6cc39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a585807 >2018-05-29 04:25:01,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [bf6cc39] FINISH, GetHardwareInfoVDSCommand, log id: 1a585807 >2018-05-29 04:25:01,483-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [521a6fda] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:25:01,491-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [47a8acec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:25:01,497-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [aa6cdb4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:25:01,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [aa6cdb4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 176d4074 >2018-05-29 04:25:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:25:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [aa6cdb4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 176d4074 >2018-05-29 04:25:01,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:01,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:01,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:01,891-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:01,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:25:01,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [aa6cdb4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:25:01,898-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [aa6cdb4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:25:01,901-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [aa6cdb4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: aa6cdb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:25:01,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 39416bb8 >2018-05-29 04:25:01,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:25:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:25:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:25:02,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:25:02,015-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:25:02,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 39416bb8 >2018-05-29 04:25:02,015-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:25:02,029-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7e33e3e8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:25:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e33e3e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656ba454 >2018-05-29 04:25:02,032-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e33e3e8] FINISH, SetVdsStatusVDSCommand, log id: 656ba454 >2018-05-29 04:25:02,064-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e33e3e8, Job ID: 1f2da9c8-2ca0-49ec-b1b8-feb08f492f8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:25:02,068-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47a8acec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:25:02,071-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7e33e3e8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47a8acec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:25:02,082-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4f1401ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:25:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4f1401ef] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5229d1] Autorecovering 2 hosts >2018-05-29 04:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5229d1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:30:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18669341] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13fcc77f >2018-05-29 04:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18669341] FINISH, SetVdsStatusVDSCommand, log id: 13fcc77f >2018-05-29 04:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [18669341] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:30:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [18669341] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:30:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41ece154 >2018-05-29 04:30:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] FINISH, SetVdsStatusVDSCommand, log id: 41ece154 >2018-05-29 04:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 28dbea73 >2018-05-29 04:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3c3b7b90] FINISH, SetHaMaintenanceModeVDSCommand, log id: 28dbea73 >2018-05-29 04:30:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3c3b7b90] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:30:01,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [23039d7c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5fafdc11 >2018-05-29 04:30:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [23039d7c] FINISH, GetHardwareInfoVDSCommand, log id: 5fafdc11 >2018-05-29 04:30:01,654-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [1a6fcb0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:30:01,658-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5a06324] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:30:01,664-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5ece490e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:30:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5ece490e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f05ea34 >2018-05-29 04:30:02,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:30:02,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5ece490e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f05ea34 >2018-05-29 04:30:02,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:02,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:02,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:02,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:02,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:02,119-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5ece490e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:02,121-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5ece490e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:30:02,124-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ece490e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ece490e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:30:02,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f02d19e >2018-05-29 04:30:02,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:30:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:30:02,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:30:02,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:30:02,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:30:02,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f02d19e >2018-05-29 04:30:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:30:02,243-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [68fbaca9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:30:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68fbaca9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 245c438e >2018-05-29 04:30:02,246-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68fbaca9] FINISH, SetVdsStatusVDSCommand, log id: 245c438e >2018-05-29 04:30:02,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 68fbaca9, Job ID: ed2e95cc-38fd-4548-b088-d8eb060098dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:30:02,259-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a06324, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:30:02,262-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68fbaca9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a06324, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:30:02,273-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [e4c2860] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:30:02,274-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [e4c2860] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:30:02,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [69dd522c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2c6f396a >2018-05-29 04:30:02,770-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [69dd522c] FINISH, GetHardwareInfoVDSCommand, log id: 2c6f396a >2018-05-29 04:30:02,783-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [435f4b96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:30:02,787-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3d5ad063] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:30:02,792-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7fe7bf23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:30:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7fe7bf23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5648e058 >2018-05-29 04:30:03,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:30:03,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7fe7bf23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5648e058 >2018-05-29 04:30:03,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:03,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:03,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:03,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:03,179-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:30:03,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7fe7bf23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:30:03,183-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7fe7bf23] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:30:03,185-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fe7bf23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fe7bf23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:30:03,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31fcab85 >2018-05-29 04:30:03,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:30:03,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:30:03,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:30:03,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31fcab85 >2018-05-29 04:30:03,285-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:30:03,298-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7e937646] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:30:03,299-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e937646] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ba1459 >2018-05-29 04:30:03,301-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e937646] FINISH, SetVdsStatusVDSCommand, log id: 6ba1459 >2018-05-29 04:30:03,306-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e937646, Job ID: 8d6f7659-2055-4a66-ba48-3a840f7bb0b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:30:03,311-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d5ad063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:30:03,313-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e937646] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d5ad063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:30:03,325-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [10f3d4a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:30:03,326-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [10f3d4a5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [26439ad0] Autorecovering 2 hosts >2018-05-29 04:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [26439ad0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [17d9eb9d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c0d7341 >2018-05-29 04:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [17d9eb9d] FINISH, SetVdsStatusVDSCommand, log id: 4c0d7341 >2018-05-29 04:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [17d9eb9d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:35:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [17d9eb9d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [b808a06] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d24bae3 >2018-05-29 04:35:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [b808a06] FINISH, SetVdsStatusVDSCommand, log id: 6d24bae3 >2018-05-29 04:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [b808a06] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2879b908 >2018-05-29 04:35:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [b808a06] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2879b908 >2018-05-29 04:35:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:35:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [b808a06] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:35:01,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5d81a297] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b9b39f3 >2018-05-29 04:35:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5d81a297] FINISH, GetHardwareInfoVDSCommand, log id: 6b9b39f3 >2018-05-29 04:35:01,037-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [4b4bee7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:35:01,046-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5e057866] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:35:01,050-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [63a0e423] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:35:01,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [63a0e423] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78073daa >2018-05-29 04:35:01,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:35:01,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [63a0e423] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78073daa >2018-05-29 04:35:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:01,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:01,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:01,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [63a0e423] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:01,444-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [63a0e423] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:35:01,447-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [63a0e423] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63a0e423, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:35:01,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5905acae >2018-05-29 04:35:01,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:35:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:35:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:35:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5905acae >2018-05-29 04:35:01,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:35:01,561-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f832f6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:35:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f832f6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2572afd9 >2018-05-29 04:35:01,564-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f832f6c] FINISH, SetVdsStatusVDSCommand, log id: 2572afd9 >2018-05-29 04:35:01,593-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f832f6c, Job ID: 10f74cf0-bb9d-4f71-bf7c-7a77ac6e83ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:35:01,597-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e057866, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:35:01,599-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f832f6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e057866, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:35:01,610-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [f8c7aad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:35:01,611-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [f8c7aad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:35:02,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3eb26dba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f7e063d >2018-05-29 04:35:02,995-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3eb26dba] FINISH, GetHardwareInfoVDSCommand, log id: 6f7e063d >2018-05-29 04:35:03,008-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3e2eec66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:35:03,012-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6a8bb8dd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:35:03,017-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5f78d927] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:35:03,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5f78d927] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6d1653a0 >2018-05-29 04:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:35:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5f78d927] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6d1653a0 >2018-05-29 04:35:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:03,412-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:03,413-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:03,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:35:03,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5f78d927] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:35:03,421-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5f78d927] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:35:03,423-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5f78d927] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f78d927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:35:03,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1e972f23 >2018-05-29 04:35:03,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1e972f23 >2018-05-29 04:35:03,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:35:03,535-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [799c53d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:35:03,537-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [799c53d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dbf42cb >2018-05-29 04:35:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [799c53d4] FINISH, SetVdsStatusVDSCommand, log id: 7dbf42cb >2018-05-29 04:35:03,544-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 799c53d4, Job ID: 15ca371e-7dc8-454d-9bea-a21e1b24898b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:35:03,549-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a8bb8dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:35:03,551-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [799c53d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a8bb8dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:35:03,561-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3a74569c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:35:03,562-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3a74569c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [57f55da8] Autorecovering 2 hosts >2018-05-29 04:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [57f55da8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70238e63] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2a709f70 >2018-05-29 04:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70238e63] FINISH, SetVdsStatusVDSCommand, log id: 2a709f70 >2018-05-29 04:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70238e63] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [70238e63] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [675b48d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43aa65b4 >2018-05-29 04:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [675b48d7] FINISH, SetVdsStatusVDSCommand, log id: 43aa65b4 >2018-05-29 04:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [675b48d7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 389fc14 >2018-05-29 04:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [675b48d7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 389fc14 >2018-05-29 04:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [675b48d7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:40:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d3cdb4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c361aea >2018-05-29 04:40:01,204-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [39d3cdb4] FINISH, GetHardwareInfoVDSCommand, log id: 7c361aea >2018-05-29 04:40:01,217-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [3e65d06d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:40:01,226-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [380fc48d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:40:01,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4fec3ad9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:40:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4fec3ad9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41c165be >2018-05-29 04:40:01,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:40:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4fec3ad9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41c165be >2018-05-29 04:40:01,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:01,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:01,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:01,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:01,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:01,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4fec3ad9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:01,690-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4fec3ad9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:40:01,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4fec3ad9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4fec3ad9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:40:01,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b21df06 >2018-05-29 04:40:01,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:40:01,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:40:01,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:40:01,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:40:01,803-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:40:01,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b21df06 >2018-05-29 04:40:01,803-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:40:01,817-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [47354a18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:40:01,818-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [47354a18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55bdfb88 >2018-05-29 04:40:01,820-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [47354a18] FINISH, SetVdsStatusVDSCommand, log id: 55bdfb88 >2018-05-29 04:40:01,852-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 47354a18, Job ID: 6c03411b-d7f2-43ca-b0a3-a41806b534cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:40:01,857-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 380fc48d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:40:01,859-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47354a18] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 380fc48d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:40:01,871-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [1c104a1f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:40:01,872-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [1c104a1f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:40:02,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [9091bce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 436b06b1 >2018-05-29 04:40:02,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [9091bce] FINISH, GetHardwareInfoVDSCommand, log id: 436b06b1 >2018-05-29 04:40:02,404-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2b1528a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:40:02,412-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1015b461] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:40:02,417-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6eceb310] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:40:02,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6eceb310] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30e6fa96 >2018-05-29 04:40:02,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:40:02,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6eceb310] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30e6fa96 >2018-05-29 04:40:02,808-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:02,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:02,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:02,813-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:02,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:40:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6eceb310] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:40:02,820-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6eceb310] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:40:02,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6eceb310] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6eceb310, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:40:02,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c0ec71 >2018-05-29 04:40:02,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:40:02,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:40:02,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:40:02,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c0ec71 >2018-05-29 04:40:02,922-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:40:02,934-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1287b61e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:40:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1287b61e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a90e9e4 >2018-05-29 04:40:02,938-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1287b61e] FINISH, SetVdsStatusVDSCommand, log id: 5a90e9e4 >2018-05-29 04:40:02,944-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1287b61e, Job ID: 1f527f82-57af-4c9b-8ec4-195812690328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:40:02,948-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1015b461, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:40:02,951-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1287b61e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1015b461, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:40:02,962-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4b795acd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:40:02,963-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4b795acd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac0a58a] Autorecovering 2 hosts >2018-05-29 04:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac0a58a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:45:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71ffeccf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75a9f62b >2018-05-29 04:45:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71ffeccf] FINISH, SetVdsStatusVDSCommand, log id: 75a9f62b >2018-05-29 04:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [71ffeccf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:45:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [71ffeccf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:45:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [589666c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b37bf77 >2018-05-29 04:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [589666c7] FINISH, SetVdsStatusVDSCommand, log id: 1b37bf77 >2018-05-29 04:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [589666c7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c108ac7 >2018-05-29 04:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [589666c7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c108ac7 >2018-05-29 04:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:45:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [589666c7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:45:00,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ebe1c89] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b1108fc >2018-05-29 04:45:00,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ebe1c89] FINISH, GetHardwareInfoVDSCommand, log id: 4b1108fc >2018-05-29 04:45:00,651-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6a7dd307] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:45:00,659-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6de9922f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:45:00,664-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7d34dc20] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:45:00,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7d34dc20] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32479cc6 >2018-05-29 04:45:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:45:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7d34dc20] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32479cc6 >2018-05-29 04:45:01,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:01,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:01,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:01,108-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:01,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:01,113-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7d34dc20] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:01,115-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7d34dc20] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:45:01,118-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7d34dc20] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d34dc20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:45:01,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77c434ef >2018-05-29 04:45:01,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:45:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:45:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:45:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77c434ef >2018-05-29 04:45:01,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:45:01,236-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1e39fa1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:45:01,238-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1e39fa1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38f89602 >2018-05-29 04:45:01,239-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1e39fa1] FINISH, SetVdsStatusVDSCommand, log id: 38f89602 >2018-05-29 04:45:01,270-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e39fa1, Job ID: 9c8524c4-4f8e-426c-b161-47d9c3390f26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:45:01,275-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6de9922f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:45:01,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1e39fa1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6de9922f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:45:01,287-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53ca11ff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:45:01,288-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53ca11ff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:45:02,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [400aa3da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e3feae0 >2018-05-29 04:45:02,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [400aa3da] FINISH, GetHardwareInfoVDSCommand, log id: 2e3feae0 >2018-05-29 04:45:02,550-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [26aeb4b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:45:02,558-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [79ca6b2b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:45:02,563-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [290556f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:45:02,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [290556f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21d650b5 >2018-05-29 04:45:02,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:45:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [290556f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21d650b5 >2018-05-29 04:45:02,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:02,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:02,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:02,971-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:02,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:45:02,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [290556f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:45:02,978-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [290556f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:45:02,981-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [290556f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 290556f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:45:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f0e1d99 >2018-05-29 04:45:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:45:03,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:45:03,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:45:03,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: f0e1d99 >2018-05-29 04:45:03,087-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:45:03,100-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [40e46c43] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:45:03,101-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40e46c43] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d0bc469 >2018-05-29 04:45:03,103-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40e46c43] FINISH, SetVdsStatusVDSCommand, log id: 5d0bc469 >2018-05-29 04:45:03,136-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40e46c43, Job ID: 17bdb476-0199-4852-bc3a-d0aea059b904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:45:03,141-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79ca6b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:45:03,143-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [40e46c43] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79ca6b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:45:03,155-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [309396e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:45:03,156-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [309396e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521dbf19] Autorecovering 2 hosts >2018-05-29 04:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521dbf19] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:50:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2c8a5a86] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 580beeab >2018-05-29 04:50:00,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2c8a5a86] FINISH, SetVdsStatusVDSCommand, log id: 580beeab >2018-05-29 04:50:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2c8a5a86] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:50:00,057-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c8a5a86] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f7acc6a >2018-05-29 04:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] FINISH, SetVdsStatusVDSCommand, log id: f7acc6a >2018-05-29 04:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b2f4f10 >2018-05-29 04:50:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3f4c782d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b2f4f10 >2018-05-29 04:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:50:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3f4c782d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:50:00,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12540dbf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1acc8e50 >2018-05-29 04:50:00,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12540dbf] FINISH, GetHardwareInfoVDSCommand, log id: 1acc8e50 >2018-05-29 04:50:00,760-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [17aee9c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:50:00,769-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1735b4ea] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:50:00,774-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [53b70db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:50:00,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [53b70db] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51557b14 >2018-05-29 04:50:01,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:50:01,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [53b70db] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51557b14 >2018-05-29 04:50:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:01,156-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:01,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:01,160-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:01,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:01,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [53b70db] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:01,169-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [53b70db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:50:01,171-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [53b70db] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53b70db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:50:01,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 321f3d6 >2018-05-29 04:50:01,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:50:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:50:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:50:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 321f3d6 >2018-05-29 04:50:01,270-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:50:01,284-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [69aa852c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:50:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [69aa852c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a43be43 >2018-05-29 04:50:01,288-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [69aa852c] FINISH, SetVdsStatusVDSCommand, log id: a43be43 >2018-05-29 04:50:01,316-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69aa852c, Job ID: 6f5b7763-9c5e-450d-a5f9-0e9c59f94078, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:50:01,321-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1735b4ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:50:01,323-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [69aa852c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1735b4ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:50:01,334-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [52aabb80] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:50:01,335-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [52aabb80] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:50:01,980-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5ffc834f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5cc98090 >2018-05-29 04:50:01,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5ffc834f] FINISH, GetHardwareInfoVDSCommand, log id: 5cc98090 >2018-05-29 04:50:01,996-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [c8fbf0c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:50:02,005-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7fbed239] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:50:02,010-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27ba0d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:50:02,022-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27ba0d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 380ee207 >2018-05-29 04:50:03,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:50:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27ba0d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 380ee207 >2018-05-29 04:50:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:03,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:03,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:03,412-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:50:03,413-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27ba0d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:50:03,417-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27ba0d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:50:03,419-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27ba0d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27ba0d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:50:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1ec06fb >2018-05-29 04:50:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:50:03,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:50:03,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:50:03,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1ec06fb >2018-05-29 04:50:03,524-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:50:03,539-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [542b46fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:50:03,541-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [542b46fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cd03692 >2018-05-29 04:50:03,543-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [542b46fa] FINISH, SetVdsStatusVDSCommand, log id: cd03692 >2018-05-29 04:50:03,577-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 542b46fa, Job ID: 9d312c4a-0329-471c-bba2-23c8dc57e069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:50:03,582-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7fbed239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:50:03,584-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [542b46fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7fbed239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:50:03,596-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2b095b34] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:50:03,598-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2b095b34] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:54:42,824-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [6a6b9572] Attempting to update VMs/Templates Ovf. >2018-05-29 04:54:42,826-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 04:54:42,826-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 04:54:42,827-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:54:42,876-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 04:54:42,880-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 04:54:42,880-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 04:54:42,880-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 04:54:42,880-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 04:54:42,880-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 04:54:42,881-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [7899619d] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 04:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [14800dac] Autorecovering 2 hosts >2018-05-29 04:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [14800dac] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:55:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51d55153] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47299f83 >2018-05-29 04:55:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51d55153] FINISH, SetVdsStatusVDSCommand, log id: 47299f83 >2018-05-29 04:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [51d55153] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 04:55:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [51d55153] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 04:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e0eee2 >2018-05-29 04:55:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] FINISH, SetVdsStatusVDSCommand, log id: 54e0eee2 >2018-05-29 04:55:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5af0f76e >2018-05-29 04:55:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cbbfcfd] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5af0f76e >2018-05-29 04:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 04:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cbbfcfd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 04:55:01,290-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [4f0d183] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2b1091ed >2018-05-29 04:55:01,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [4f0d183] FINISH, GetHardwareInfoVDSCommand, log id: 2b1091ed >2018-05-29 04:55:01,304-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3f47b872] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:55:01,308-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2a85646a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:55:01,313-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [713c8fd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:55:01,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [713c8fd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fdc8490 >2018-05-29 04:55:01,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:55:01,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [713c8fd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fdc8490 >2018-05-29 04:55:01,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:01,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:01,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:01,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [713c8fd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:01,771-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [713c8fd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:55:01,774-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [713c8fd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 713c8fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:55:01,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18531e71 >2018-05-29 04:55:01,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:55:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:55:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:55:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18531e71 >2018-05-29 04:55:01,881-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:55:01,895-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [7140868f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:55:01,897-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7140868f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29f58023 >2018-05-29 04:55:01,899-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7140868f] FINISH, SetVdsStatusVDSCommand, log id: 29f58023 >2018-05-29 04:55:01,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [487f0b56] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: e38365d >2018-05-29 04:55:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [487f0b56] FINISH, GetHardwareInfoVDSCommand, log id: e38365d >2018-05-29 04:55:01,914-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7140868f, Job ID: a32233d3-8fad-45e7-8f51-4cd0f3f5e1c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:55:01,918-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [29c3a564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:55:01,920-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2a85646a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:55:01,923-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7140868f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2a85646a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:55:01,924-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15aac258] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:55:01,931-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55ac5bac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 04:55:01,939-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [499c9fd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 04:55:01,941-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [499c9fd7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 04:55:01,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55ac5bac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f3704b3 >2018-05-29 04:55:02,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 04:55:02,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55ac5bac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f3704b3 >2018-05-29 04:55:02,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:02,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:02,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:02,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:02,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 04:55:02,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55ac5bac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 04:55:02,343-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55ac5bac] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 04:55:02,346-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55ac5bac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55ac5bac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 04:55:02,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a93cdaf >2018-05-29 04:55:02,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:55:02,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 04:55:02,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 04:55:02,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a93cdaf >2018-05-29 04:55:02,448-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 04:55:02,460-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3af2c9d0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:55:02,462-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3af2c9d0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 522294a7 >2018-05-29 04:55:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3af2c9d0] FINISH, SetVdsStatusVDSCommand, log id: 522294a7 >2018-05-29 04:55:02,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3af2c9d0, Job ID: b6ce40cf-06ca-445c-95ad-104699610c9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 04:55:02,474-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15aac258, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 04:55:02,476-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3af2c9d0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15aac258, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 04:55:02,487-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [59d1d37f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 04:55:02,488-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [59d1d37f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:00:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [63a320a2] Autorecovering 2 hosts >2018-05-29 05:00:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [63a320a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4c750321] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 203f92fb >2018-05-29 05:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4c750321] FINISH, SetVdsStatusVDSCommand, log id: 203f92fb >2018-05-29 05:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4c750321] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c750321] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3605089c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51bce765 >2018-05-29 05:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3605089c] FINISH, SetVdsStatusVDSCommand, log id: 51bce765 >2018-05-29 05:00:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3605089c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 599f5a01 >2018-05-29 05:00:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3605089c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 599f5a01 >2018-05-29 05:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:00:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3605089c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:00:02,726-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [fc16e08] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 315ed790 >2018-05-29 05:00:02,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [fc16e08] FINISH, GetHardwareInfoVDSCommand, log id: 315ed790 >2018-05-29 05:00:02,742-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [25325821] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:00:02,751-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5f7fbfc8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:00:02,756-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [195e613b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:00:02,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [195e613b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e91be66 >2018-05-29 05:00:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7542ca35] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e643ae7 >2018-05-29 05:00:03,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7542ca35] FINISH, GetHardwareInfoVDSCommand, log id: 7e643ae7 >2018-05-29 05:00:03,088-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6bea44e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:00:03,093-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6fb48f07] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:00:03,098-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [394a52cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:00:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [394a52cc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 492ef51a >2018-05-29 05:00:03,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:00:03,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [195e613b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e91be66 >2018-05-29 05:00:03,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,148-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [195e613b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,151-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [195e613b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:00:03,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [195e613b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 195e613b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:00:03,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2be26b29 >2018-05-29 05:00:03,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:00:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:00:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:00:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2be26b29 >2018-05-29 05:00:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:00:03,270-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [45e34b3d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:00:03,271-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45e34b3d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f1779d1 >2018-05-29 05:00:03,273-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45e34b3d] FINISH, SetVdsStatusVDSCommand, log id: 6f1779d1 >2018-05-29 05:00:03,305-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45e34b3d, Job ID: 09dae4a3-fbe2-4fe3-9dff-5ef54b753764, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:00:03,309-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f7fbfc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:00:03,311-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [45e34b3d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f7fbfc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:00:03,323-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [591ab818] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:00:03,324-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [591ab818] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:00:03,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:00:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [394a52cc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 492ef51a >2018-05-29 05:00:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,486-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:00:03,495-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [394a52cc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:00:03,498-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [394a52cc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:00:03,500-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [394a52cc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 394a52cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:00:03,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a0ebf03 >2018-05-29 05:00:03,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:00:03,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:00:03,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:00:03,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a0ebf03 >2018-05-29 05:00:03,602-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:00:03,615-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [4d8d020] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:00:03,616-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d8d020] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e79574f >2018-05-29 05:00:03,619-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d8d020] FINISH, SetVdsStatusVDSCommand, log id: 5e79574f >2018-05-29 05:00:03,624-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d8d020, Job ID: dd3f0010-6ba3-4ece-9649-6b1b2722ee7c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:00:03,628-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fb48f07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:00:03,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4d8d020] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fb48f07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:00:03,641-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [634498af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:00:03,642-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [634498af] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d8b11d6] Autorecovering 2 hosts >2018-05-29 05:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d8b11d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:05:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [76701550] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47f67484 >2018-05-29 05:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [76701550] FINISH, SetVdsStatusVDSCommand, log id: 47f67484 >2018-05-29 05:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [76701550] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:05:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [76701550] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26be8864 >2018-05-29 05:05:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] FINISH, SetVdsStatusVDSCommand, log id: 26be8864 >2018-05-29 05:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58320987 >2018-05-29 05:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6d6fa1a3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58320987 >2018-05-29 05:05:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d6fa1a3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:05:00,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [64f7e9a7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42121973 >2018-05-29 05:05:00,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [64f7e9a7] FINISH, GetHardwareInfoVDSCommand, log id: 42121973 >2018-05-29 05:05:00,919-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [706e4df1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:05:00,928-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [b395b90] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:05:00,933-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [25b214a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:05:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [25b214a8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b42fab1 >2018-05-29 05:05:01,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3af8a134] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a068ef1 >2018-05-29 05:05:01,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3af8a134] FINISH, GetHardwareInfoVDSCommand, log id: 2a068ef1 >2018-05-29 05:05:01,208-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3fd4eb71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:05:01,212-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [19fd695a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:05:01,217-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [41d25aaa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:05:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [41d25aaa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7cd73fc6 >2018-05-29 05:05:01,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:05:01,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [25b214a8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b42fab1 >2018-05-29 05:05:01,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,309-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [25b214a8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,320-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [25b214a8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:05:01,323-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25b214a8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 25b214a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:05:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f0d96bb >2018-05-29 05:05:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:05:01,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:05:01,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:05:01,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: f0d96bb >2018-05-29 05:05:01,424-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:05:01,437-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4fd85c0c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:05:01,439-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4fd85c0c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fb36774 >2018-05-29 05:05:01,441-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4fd85c0c] FINISH, SetVdsStatusVDSCommand, log id: fb36774 >2018-05-29 05:05:01,471-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fd85c0c, Job ID: ac0224b8-9c85-4b71-a4c7-56c0e820f31e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:05:01,476-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b395b90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:05:01,479-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4fd85c0c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b395b90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:05:01,490-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3b1fb3d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:05:01,491-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3b1fb3d0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:05:01,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:05:01,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [41d25aaa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7cd73fc6 >2018-05-29 05:05:01,612-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,616-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,617-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:05:01,621-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [41d25aaa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:05:01,625-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [41d25aaa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:05:01,628-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [41d25aaa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41d25aaa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:05:01,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f239aeb >2018-05-29 05:05:01,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:05:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:05:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:05:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f239aeb >2018-05-29 05:05:01,732-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:05:01,746-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7633c97] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:05:01,747-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7633c97] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cb823ec >2018-05-29 05:05:01,749-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7633c97] FINISH, SetVdsStatusVDSCommand, log id: 7cb823ec >2018-05-29 05:05:01,755-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7633c97, Job ID: 271bf38d-3f9e-4056-9a69-c7d451e11f9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:05:01,760-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19fd695a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:05:01,762-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7633c97] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19fd695a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:05:01,773-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5a1336bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:05:01,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5a1336bd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [423e1413] Autorecovering 2 hosts >2018-05-29 05:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [423e1413] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a4d302c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b55b00f >2018-05-29 05:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a4d302c] FINISH, SetVdsStatusVDSCommand, log id: 4b55b00f >2018-05-29 05:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2a4d302c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2a4d302c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eae8c60 >2018-05-29 05:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] FINISH, SetVdsStatusVDSCommand, log id: 2eae8c60 >2018-05-29 05:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 41092827 >2018-05-29 05:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2b600d2c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 41092827 >2018-05-29 05:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2b600d2c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:10:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3d9f7295] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 47099baf >2018-05-29 05:10:02,397-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3d9f7295] FINISH, GetHardwareInfoVDSCommand, log id: 47099baf >2018-05-29 05:10:02,412-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [560c7155] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:10:02,421-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2bc2eb94] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:10:02,426-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2e9a89ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:10:02,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2e9a89ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 413b8a8 >2018-05-29 05:10:02,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [eaaf1ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 665bf7ca >2018-05-29 05:10:02,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [eaaf1ac] FINISH, GetHardwareInfoVDSCommand, log id: 665bf7ca >2018-05-29 05:10:02,674-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5cef15ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:10:02,678-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [399de8b6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:10:02,683-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [16a7b04b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:10:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [16a7b04b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30c2c2e3 >2018-05-29 05:10:03,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:10:03,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [16a7b04b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30c2c2e3 >2018-05-29 05:10:03,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [16a7b04b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,089-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [16a7b04b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:10:03,091-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [16a7b04b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 16a7b04b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:10:03,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44a989e5 >2018-05-29 05:10:03,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:10:03,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:10:03,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:10:03,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44a989e5 >2018-05-29 05:10:03,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:10:03,210-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3a9e58ad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:10:03,212-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3a9e58ad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a722bcd >2018-05-29 05:10:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3a9e58ad] FINISH, SetVdsStatusVDSCommand, log id: a722bcd >2018-05-29 05:10:03,246-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3a9e58ad, Job ID: 0dfcefcc-70ec-4cc0-a4d4-0b9f00c5beb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:10:03,251-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 399de8b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:10:03,253-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3a9e58ad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 399de8b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:10:03,264-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2ce9e805] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:10:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2ce9e805] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:10:03,850-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:10:03,850-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2e9a89ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 413b8a8 >2018-05-29 05:10:03,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,855-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,860-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:10:03,864-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2e9a89ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:10:03,867-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2e9a89ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:10:03,870-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e9a89ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2e9a89ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:10:03,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c20521f >2018-05-29 05:10:03,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:10:03,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:10:03,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:10:03,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c20521f >2018-05-29 05:10:03,968-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:10:03,981-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2fa2f736] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:10:03,982-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fa2f736] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b3ebca4 >2018-05-29 05:10:03,984-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fa2f736] FINISH, SetVdsStatusVDSCommand, log id: 5b3ebca4 >2018-05-29 05:10:03,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fa2f736, Job ID: ce3926e7-b017-40fd-8a52-980fa93282b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:10:03,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bc2eb94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:10:03,996-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fa2f736] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bc2eb94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:10:04,007-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2c1b62aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:10:04,008-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2c1b62aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [25412e13] Autorecovering 2 hosts >2018-05-29 05:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [25412e13] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:15:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [950b1e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7293ea2a >2018-05-29 05:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [950b1e2] FINISH, SetVdsStatusVDSCommand, log id: 7293ea2a >2018-05-29 05:15:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [950b1e2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:15:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [950b1e2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:15:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27abd083 >2018-05-29 05:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] FINISH, SetVdsStatusVDSCommand, log id: 27abd083 >2018-05-29 05:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 599a3009 >2018-05-29 05:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e9f7f53] FINISH, SetHaMaintenanceModeVDSCommand, log id: 599a3009 >2018-05-29 05:15:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e9f7f53] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:15:00,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b8a0475] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ee5f3f3 >2018-05-29 05:15:00,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b8a0475] FINISH, GetHardwareInfoVDSCommand, log id: 1ee5f3f3 >2018-05-29 05:15:00,874-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7f0b5c33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:15:00,882-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3b362af5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:15:00,887-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5215ffa8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:15:00,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5215ffa8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65dd5eb0 >2018-05-29 05:15:01,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:15:01,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5215ffa8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65dd5eb0 >2018-05-29 05:15:01,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:01,272-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:01,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:01,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:01,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:01,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5215ffa8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:01,285-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5215ffa8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:15:01,288-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5215ffa8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5215ffa8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:15:01,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36fcb20b >2018-05-29 05:15:01,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:15:01,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:15:01,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:15:01,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36fcb20b >2018-05-29 05:15:01,390-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:15:01,404-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [71b45dcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:15:01,405-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71b45dcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ccf0dd8 >2018-05-29 05:15:01,407-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71b45dcc] FINISH, SetVdsStatusVDSCommand, log id: 6ccf0dd8 >2018-05-29 05:15:01,436-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71b45dcc, Job ID: 57264229-0e0d-48a5-b80c-4f8421630cd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:15:01,441-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b362af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:15:01,443-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71b45dcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b362af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:15:01,453-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2fd51123] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:15:01,454-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2fd51123] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:15:01,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e03f5f0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 492e0e55 >2018-05-29 05:15:01,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e03f5f0] FINISH, GetHardwareInfoVDSCommand, log id: 492e0e55 >2018-05-29 05:15:01,666-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4309e266] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:15:01,670-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7e5a979f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:15:01,676-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b0548cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:15:01,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b0548cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33dfbf39 >2018-05-29 05:15:02,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:15:02,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b0548cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33dfbf39 >2018-05-29 05:15:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:02,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:02,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:02,063-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:02,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:15:02,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b0548cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:15:02,070-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b0548cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:15:02,073-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b0548cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b0548cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:15:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d17cd8d >2018-05-29 05:15:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:15:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:15:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:15:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d17cd8d >2018-05-29 05:15:02,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:15:02,191-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [aeefeb2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:15:02,193-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [aeefeb2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bdbd16 >2018-05-29 05:15:02,195-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [aeefeb2] FINISH, SetVdsStatusVDSCommand, log id: bdbd16 >2018-05-29 05:15:02,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: aeefeb2, Job ID: 6dba3f8f-1f78-457d-90a6-d0178002a89d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:15:02,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e5a979f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:15:02,234-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [aeefeb2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e5a979f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:15:02,244-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5fc2b209] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:15:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5fc2b209] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a7a341d] Autorecovering 2 hosts >2018-05-29 05:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a7a341d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:20:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79bdcacf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3153f99d >2018-05-29 05:20:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79bdcacf] FINISH, SetVdsStatusVDSCommand, log id: 3153f99d >2018-05-29 05:20:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79bdcacf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:20:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79bdcacf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:20:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:20:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:20:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4564d76b >2018-05-29 05:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] FINISH, SetVdsStatusVDSCommand, log id: 4564d76b >2018-05-29 05:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d788f63 >2018-05-29 05:20:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6ae4e99f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d788f63 >2018-05-29 05:20:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:20:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6ae4e99f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:20:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [271a2d48] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4f36a2d5 >2018-05-29 05:20:02,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [271a2d48] FINISH, GetHardwareInfoVDSCommand, log id: 4f36a2d5 >2018-05-29 05:20:02,057-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4ac3821f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:20:02,062-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [76ad38ce] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:20:02,067-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [daf0038] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:20:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [daf0038] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 507c3af6 >2018-05-29 05:20:02,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:20:02,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [daf0038] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 507c3af6 >2018-05-29 05:20:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:02,464-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:02,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:02,473-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [daf0038] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:02,476-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [daf0038] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:20:02,478-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [daf0038] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: daf0038, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:20:02,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18d8a13d >2018-05-29 05:20:02,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:20:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:20:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:20:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18d8a13d >2018-05-29 05:20:02,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:20:02,595-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6c4b9d16] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:20:02,596-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6c4b9d16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b9f9b1d >2018-05-29 05:20:02,598-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6c4b9d16] FINISH, SetVdsStatusVDSCommand, log id: 3b9f9b1d >2018-05-29 05:20:02,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6c4b9d16, Job ID: 71f0f28c-f390-49bc-a6dd-af34505425f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:20:02,611-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 76ad38ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:20:02,613-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6c4b9d16] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 76ad38ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:20:02,624-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [308acb46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:20:02,625-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [308acb46] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:20:02,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5ddc00b3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 598de4c6 >2018-05-29 05:20:02,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5ddc00b3] FINISH, GetHardwareInfoVDSCommand, log id: 598de4c6 >2018-05-29 05:20:02,888-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [27032220] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:20:02,893-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1360f123] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:20:02,898-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2b8c48e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:20:02,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2b8c48e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 199cf55b >2018-05-29 05:20:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:20:04,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2b8c48e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 199cf55b >2018-05-29 05:20:04,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:04,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:04,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:04,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:04,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:20:04,251-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2b8c48e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:20:04,255-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2b8c48e4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:20:04,258-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2b8c48e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2b8c48e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:20:04,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 27c2e411 >2018-05-29 05:20:04,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:20:04,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:20:04,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:20:04,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:20:04,359-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:20:04,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 27c2e411 >2018-05-29 05:20:04,359-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:20:04,372-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [360290d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:20:04,373-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [360290d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7908cb2a >2018-05-29 05:20:04,375-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [360290d2] FINISH, SetVdsStatusVDSCommand, log id: 7908cb2a >2018-05-29 05:20:04,381-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 360290d2, Job ID: 4daa7888-7be8-4e65-8aec-5c9ef4bef231, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:20:04,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1360f123, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:20:04,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [360290d2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1360f123, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:20:04,398-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [291b67a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:20:04,399-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [291b67a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d2c2c7a] Autorecovering 2 hosts >2018-05-29 05:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d2c2c7a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:25:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a8de19a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f808a3 >2018-05-29 05:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a8de19a] FINISH, SetVdsStatusVDSCommand, log id: 5f808a3 >2018-05-29 05:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4a8de19a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:25:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4a8de19a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17019fc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 606a9d01 >2018-05-29 05:25:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17019fc5] FINISH, SetVdsStatusVDSCommand, log id: 606a9d01 >2018-05-29 05:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17019fc5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79357741 >2018-05-29 05:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17019fc5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79357741 >2018-05-29 05:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:25:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17019fc5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:25:00,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79df7166] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60373059 >2018-05-29 05:25:00,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79df7166] FINISH, GetHardwareInfoVDSCommand, log id: 60373059 >2018-05-29 05:25:00,302-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [51811aff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:25:00,311-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3f03e5e5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:25:00,316-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2c47e59b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:25:00,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2c47e59b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f9c59f6 >2018-05-29 05:25:00,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:25:00,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2c47e59b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f9c59f6 >2018-05-29 05:25:00,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:00,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:00,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:00,766-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:00,770-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:00,771-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2c47e59b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:00,773-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2c47e59b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:25:00,776-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c47e59b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c47e59b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:25:00,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19759c9e >2018-05-29 05:25:00,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:25:00,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:25:00,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:25:00,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19759c9e >2018-05-29 05:25:00,882-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:25:00,895-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4006b18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:25:00,897-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4006b18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f326c7e >2018-05-29 05:25:00,899-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4006b18] FINISH, SetVdsStatusVDSCommand, log id: 5f326c7e >2018-05-29 05:25:00,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4006b18, Job ID: 8819e555-c06e-4401-b673-b176629854ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:25:00,935-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f03e5e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:25:00,938-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4006b18] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f03e5e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:25:00,949-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [30907d38] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:25:00,950-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [30907d38] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:25:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [12cd66e2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 414b4cbe >2018-05-29 05:25:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [12cd66e2] FINISH, GetHardwareInfoVDSCommand, log id: 414b4cbe >2018-05-29 05:25:02,147-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3386a231] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:25:02,152-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4762eace] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:25:02,157-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a4d45e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:25:02,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a4d45e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ea23563 >2018-05-29 05:25:02,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:25:02,544-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a4d45e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ea23563 >2018-05-29 05:25:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:02,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:02,553-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:02,554-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:02,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:25:02,558-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a4d45e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:25:02,561-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a4d45e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:25:02,563-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a4d45e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a4d45e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:25:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52bfc385 >2018-05-29 05:25:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:25:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:25:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:25:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52bfc385 >2018-05-29 05:25:02,666-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:25:02,679-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4e10d21e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:25:02,680-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4e10d21e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21b5bfdd >2018-05-29 05:25:02,683-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4e10d21e] FINISH, SetVdsStatusVDSCommand, log id: 21b5bfdd >2018-05-29 05:25:02,688-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e10d21e, Job ID: 2aed3453-b0f6-4ad6-8319-903517acf2a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:25:02,692-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4762eace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:25:02,695-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e10d21e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4762eace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:25:02,705-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5c63ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:25:02,706-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5c63ed7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [745ce051] Autorecovering 2 hosts >2018-05-29 05:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [745ce051] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [49d07df] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b96d559 >2018-05-29 05:30:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [49d07df] FINISH, SetVdsStatusVDSCommand, log id: 2b96d559 >2018-05-29 05:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [49d07df] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:30:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49d07df] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [312ff299] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777630f0 >2018-05-29 05:30:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [312ff299] FINISH, SetVdsStatusVDSCommand, log id: 777630f0 >2018-05-29 05:30:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [312ff299] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 55673cdc >2018-05-29 05:30:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [312ff299] FINISH, SetHaMaintenanceModeVDSCommand, log id: 55673cdc >2018-05-29 05:30:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:30:00,108-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [312ff299] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 312ff299, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 05:30:00,108-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [312ff299] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:30:00,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d5ba950] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44d9afbe >2018-05-29 05:30:00,369-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d5ba950] FINISH, GetHardwareInfoVDSCommand, log id: 44d9afbe >2018-05-29 05:30:00,382-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [2c68d0aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:30:00,390-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [20c8d7af] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:30:00,396-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [45cb8c5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:30:00,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [45cb8c5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f4b1cd2 >2018-05-29 05:30:00,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:30:00,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [45cb8c5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f4b1cd2 >2018-05-29 05:30:00,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:00,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:00,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:00,797-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:00,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:00,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [45cb8c5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:00,804-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [45cb8c5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:30:00,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45cb8c5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45cb8c5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:30:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37dfaec6 >2018-05-29 05:30:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:30:00,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:30:00,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:30:00,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:30:00,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:30:00,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37dfaec6 >2018-05-29 05:30:00,927-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:30:00,940-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [78f178cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:30:00,941-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [78f178cc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225bdd0e >2018-05-29 05:30:00,943-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [78f178cc] FINISH, SetVdsStatusVDSCommand, log id: 225bdd0e >2018-05-29 05:30:00,977-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78f178cc, Job ID: 9fefc0fe-f3dc-4e63-9c57-9ccc6780c679, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:30:00,981-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20c8d7af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:30:00,984-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [78f178cc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20c8d7af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:30:00,995-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3a7607c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:30:00,996-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3a7607c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:30:01,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1ae57bec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 791f84be >2018-05-29 05:30:01,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1ae57bec] FINISH, GetHardwareInfoVDSCommand, log id: 791f84be >2018-05-29 05:30:01,543-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ee58503] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:30:01,547-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6d623fb8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:30:01,552-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1ced95e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:30:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1ced95e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f54bfc6 >2018-05-29 05:30:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:30:01,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1ced95e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f54bfc6 >2018-05-29 05:30:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:01,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:01,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:01,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:30:01,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1ced95e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:30:01,952-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1ced95e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:30:01,955-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1ced95e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ced95e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:30:01,958-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 237f2db2 >2018-05-29 05:30:01,958-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:30:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:30:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:30:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 237f2db2 >2018-05-29 05:30:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:30:02,070-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [56e0d89b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:30:02,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [56e0d89b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ce9bcd6 >2018-05-29 05:30:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [56e0d89b] FINISH, SetVdsStatusVDSCommand, log id: 5ce9bcd6 >2018-05-29 05:30:02,108-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56e0d89b, Job ID: 60832dfb-5d40-4667-ac0d-7b12fe3dde56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:30:02,112-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d623fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:30:02,114-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [56e0d89b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d623fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:30:02,125-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [1ab38b2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:30:02,126-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [1ab38b2d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [758ebd08] Autorecovering 2 hosts >2018-05-29 05:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [758ebd08] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:35:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:35:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:35:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f44b03e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a7eb3b3 >2018-05-29 05:35:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f44b03e] FINISH, SetVdsStatusVDSCommand, log id: 3a7eb3b3 >2018-05-29 05:35:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5f44b03e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:35:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5f44b03e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:35:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [180f10da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5582838d >2018-05-29 05:35:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [180f10da] FINISH, SetVdsStatusVDSCommand, log id: 5582838d >2018-05-29 05:35:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [180f10da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1b91382 >2018-05-29 05:35:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [180f10da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1b91382 >2018-05-29 05:35:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [180f10da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:35:01,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [229a9879] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2b3908ec >2018-05-29 05:35:01,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [229a9879] FINISH, GetHardwareInfoVDSCommand, log id: 2b3908ec >2018-05-29 05:35:01,675-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [757cbfcf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:35:01,679-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6bfe2a69] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:35:01,684-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [de56bd0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:35:01,743-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [de56bd0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f8c4e2e >2018-05-29 05:35:02,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:35:02,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [de56bd0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f8c4e2e >2018-05-29 05:35:02,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:02,129-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:02,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:02,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:02,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:02,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [de56bd0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:02,141-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [de56bd0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:35:02,144-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [de56bd0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: de56bd0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:35:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6275ec0e >2018-05-29 05:35:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:35:02,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:35:02,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:35:02,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6275ec0e >2018-05-29 05:35:02,248-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:35:02,261-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [61f761d7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:35:02,263-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [61f761d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62ae9350 >2018-05-29 05:35:02,265-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [61f761d7] FINISH, SetVdsStatusVDSCommand, log id: 62ae9350 >2018-05-29 05:35:02,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 61f761d7, Job ID: 5305a0be-0f0a-4f0e-baf8-dc7749c82f4c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:35:02,302-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6bfe2a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:35:02,304-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [61f761d7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6bfe2a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:35:02,316-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7aa5c841] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:35:02,317-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7aa5c841] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:35:02,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59053644] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ea56128 >2018-05-29 05:35:02,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59053644] FINISH, GetHardwareInfoVDSCommand, log id: 1ea56128 >2018-05-29 05:35:02,712-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6c7c22fe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:35:02,717-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [44f7fd3e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:35:02,722-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [334a6c83] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:35:02,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [334a6c83] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1badb7ac >2018-05-29 05:35:03,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:35:03,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [334a6c83] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1badb7ac >2018-05-29 05:35:03,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:03,113-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:03,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:03,118-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:03,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:35:03,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [334a6c83] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:35:03,125-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [334a6c83] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:35:03,128-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [334a6c83] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 334a6c83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:35:03,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66fa9acd >2018-05-29 05:35:03,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:35:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:35:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:35:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66fa9acd >2018-05-29 05:35:03,227-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:35:03,240-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [e540ccc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:35:03,242-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e540ccc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63a9b7ba >2018-05-29 05:35:03,244-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e540ccc] FINISH, SetVdsStatusVDSCommand, log id: 63a9b7ba >2018-05-29 05:35:03,249-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e540ccc, Job ID: cd07f803-50e4-4596-b8cb-61bf9885dc6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:35:03,254-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 44f7fd3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:35:03,256-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [e540ccc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 44f7fd3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:35:03,267-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [28e24d49] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:35:03,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [28e24d49] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:40:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6006be9b] Autorecovering 2 hosts >2018-05-29 05:40:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6006be9b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:40:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [45d0c99f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34efdba8 >2018-05-29 05:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [45d0c99f] FINISH, SetVdsStatusVDSCommand, log id: 34efdba8 >2018-05-29 05:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [45d0c99f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:40:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [45d0c99f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:40:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [129cf305] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25fcdced >2018-05-29 05:40:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [129cf305] FINISH, SetVdsStatusVDSCommand, log id: 25fcdced >2018-05-29 05:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [129cf305] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6b794bc9 >2018-05-29 05:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [129cf305] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6b794bc9 >2018-05-29 05:40:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:40:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [129cf305] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:40:00,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6e60e952] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c23fb68 >2018-05-29 05:40:00,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6e60e952] FINISH, GetHardwareInfoVDSCommand, log id: 7c23fb68 >2018-05-29 05:40:00,904-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6f01d241] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:40:00,912-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [48868cfa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:40:00,918-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [38462093] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:40:00,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [38462093] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b10ff3b >2018-05-29 05:40:01,307-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:40:01,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [38462093] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b10ff3b >2018-05-29 05:40:01,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:01,312-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:01,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:01,316-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:01,320-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:01,321-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [38462093] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:01,323-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [38462093] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:40:01,326-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38462093] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 38462093, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:40:01,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75855d11 >2018-05-29 05:40:01,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:40:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:40:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:40:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75855d11 >2018-05-29 05:40:01,432-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:40:01,444-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2f6dfb50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:40:01,445-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f6dfb50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f633746 >2018-05-29 05:40:01,447-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f6dfb50] FINISH, SetVdsStatusVDSCommand, log id: 2f633746 >2018-05-29 05:40:01,479-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2f6dfb50, Job ID: 9ef40f70-b9dc-42d5-9ec7-278cb9df25cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:40:01,483-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 48868cfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:40:01,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f6dfb50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 48868cfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:40:01,496-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3c0d7061] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:40:01,497-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3c0d7061] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:40:03,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2885a07b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 372a78c7 >2018-05-29 05:40:03,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2885a07b] FINISH, GetHardwareInfoVDSCommand, log id: 372a78c7 >2018-05-29 05:40:03,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3705db16] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:40:03,103-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5ea2bece] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:40:03,108-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dd2abf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:40:03,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dd2abf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7bc90c67 >2018-05-29 05:40:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dd2abf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7bc90c67 >2018-05-29 05:40:03,514-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:03,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:03,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:03,523-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:40:03,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dd2abf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:40:03,526-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dd2abf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:40:03,529-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dd2abf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51dd2abf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:40:03,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62619972 >2018-05-29 05:40:03,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:40:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:40:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:40:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62619972 >2018-05-29 05:40:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:40:03,639-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [46580a2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:40:03,640-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [46580a2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 514616ed >2018-05-29 05:40:03,642-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [46580a2c] FINISH, SetVdsStatusVDSCommand, log id: 514616ed >2018-05-29 05:40:03,673-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46580a2c, Job ID: 7fb5e6c7-c5df-49e7-a271-7bbda5526baf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:40:03,678-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ea2bece, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:40:03,680-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [46580a2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ea2bece, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:40:03,690-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [51980f46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:40:03,691-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [51980f46] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [30da610d] Autorecovering 2 hosts >2018-05-29 05:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [30da610d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52f83e5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7eaa25fe >2018-05-29 05:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52f83e5b] FINISH, SetVdsStatusVDSCommand, log id: 7eaa25fe >2018-05-29 05:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52f83e5b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [52f83e5b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:45:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72a48cbf >2018-05-29 05:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] FINISH, SetVdsStatusVDSCommand, log id: 72a48cbf >2018-05-29 05:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 31990db1 >2018-05-29 05:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5aaab1e1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 31990db1 >2018-05-29 05:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:45:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5aaab1e1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:45:01,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c8f7cda] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2e3c7c2f >2018-05-29 05:45:01,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c8f7cda] FINISH, GetHardwareInfoVDSCommand, log id: 2e3c7c2f >2018-05-29 05:45:01,367-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5e134719] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:45:01,376-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1dfcaa1c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:45:01,382-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7b673956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:45:01,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7b673956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1617786 >2018-05-29 05:45:01,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:45:01,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7b673956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1617786 >2018-05-29 05:45:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:01,765-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:01,770-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7b673956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:01,777-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7b673956] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:45:01,779-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b673956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7b673956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:45:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c404fa4 >2018-05-29 05:45:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:45:01,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:45:01,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:45:01,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c404fa4 >2018-05-29 05:45:01,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:45:01,899-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4eeeadc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:45:01,900-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4eeeadc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2e887ed8 >2018-05-29 05:45:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4eeeadc] FINISH, SetVdsStatusVDSCommand, log id: 2e887ed8 >2018-05-29 05:45:01,936-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4eeeadc, Job ID: 0d177def-7701-4b0a-aede-b1746ef079d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:45:01,940-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1dfcaa1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:45:01,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4eeeadc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1dfcaa1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:45:01,954-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [58c43eea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:45:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [58c43eea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:45:02,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [486fc4ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e2f2e2b >2018-05-29 05:45:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [486fc4ed] FINISH, GetHardwareInfoVDSCommand, log id: 3e2f2e2b >2018-05-29 05:45:02,049-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [774db8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:45:02,053-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1148001e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:45:02,059-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [67d1746e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:45:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [67d1746e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14b39cb4 >2018-05-29 05:45:02,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:45:02,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [67d1746e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14b39cb4 >2018-05-29 05:45:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:02,464-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:02,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:45:02,473-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [67d1746e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:45:02,476-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [67d1746e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:45:02,478-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [67d1746e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 67d1746e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:45:02,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c1e6eba >2018-05-29 05:45:02,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:45:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:45:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:45:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c1e6eba >2018-05-29 05:45:02,583-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:45:02,596-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6e6f5978] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:45:02,598-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e6f5978] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 68f18f45 >2018-05-29 05:45:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e6f5978] FINISH, SetVdsStatusVDSCommand, log id: 68f18f45 >2018-05-29 05:45:02,609-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e6f5978, Job ID: 82028b81-d238-46ac-9f45-d2ed843ab5ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:45:02,613-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1148001e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:45:02,615-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e6f5978] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1148001e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:45:02,627-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53cb9cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:45:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53cb9cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5831c840] Autorecovering 2 hosts >2018-05-29 05:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5831c840] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:50:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:50:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,120-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5350da13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ae1310e >2018-05-29 05:50:00,123-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5350da13] FINISH, SetVdsStatusVDSCommand, log id: 4ae1310e >2018-05-29 05:50:00,124-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,128-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5350da13] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 5350da13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 05:50:00,128-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5350da13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:50:00,128-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5350da13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:50:00,135-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:50:00,141-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:50:00,141-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,141-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,142-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18841170 >2018-05-29 05:50:00,145-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] FINISH, SetVdsStatusVDSCommand, log id: 18841170 >2018-05-29 05:50:00,147-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 397bae1a >2018-05-29 05:50:00,149-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2a4ccc14] FINISH, SetHaMaintenanceModeVDSCommand, log id: 397bae1a >2018-05-29 05:50:00,149-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:50:00,151-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a4ccc14] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:50:03,038-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1220b34e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3d4bc298 >2018-05-29 05:50:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1220b34e] FINISH, GetHardwareInfoVDSCommand, log id: 3d4bc298 >2018-05-29 05:50:03,054-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [75c43e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:50:03,063-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7f8d42d1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:50:03,070-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [18298682] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:50:03,129-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [18298682] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2bbbcdfc >2018-05-29 05:50:03,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [340320a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6466a0bd >2018-05-29 05:50:03,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [340320a5] FINISH, GetHardwareInfoVDSCommand, log id: 6466a0bd >2018-05-29 05:50:03,205-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4793517] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:50:03,210-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [e731268] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:50:03,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6e27f12d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:50:03,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6e27f12d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c06a3f >2018-05-29 05:50:03,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:50:03,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [18298682] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2bbbcdfc >2018-05-29 05:50:03,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,527-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [18298682] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,530-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [18298682] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:50:03,532-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18298682] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18298682, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:50:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2038c81f >2018-05-29 05:50:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:50:03,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:50:03,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6e27f12d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c06a3f >2018-05-29 05:50:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,604-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,609-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:50:03,614-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6e27f12d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:50:03,616-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6e27f12d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:50:03,619-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6e27f12d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6e27f12d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:50:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:50:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:50:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2038c81f >2018-05-29 05:50:03,639-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:50:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a88901 >2018-05-29 05:50:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:50:03,653-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [133086f7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:50:03,654-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [133086f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e56b1a8 >2018-05-29 05:50:03,656-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [133086f7] FINISH, SetVdsStatusVDSCommand, log id: e56b1a8 >2018-05-29 05:50:03,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 133086f7, Job ID: 84a5b2c6-02c5-443b-9068-479573be8414, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:50:03,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f8d42d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:50:03,696-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [133086f7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f8d42d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:50:03,706-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [b72a530] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:50:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [b72a530] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:50:03,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:50:03,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:50:03,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a88901 >2018-05-29 05:50:03,739-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:50:03,752-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [8149543] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:50:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [8149543] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6213560e >2018-05-29 05:50:03,756-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [8149543] FINISH, SetVdsStatusVDSCommand, log id: 6213560e >2018-05-29 05:50:03,761-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8149543, Job ID: 0bc2224b-ee2e-4247-a832-36ddac89f0bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:50:03,765-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e731268, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:50:03,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8149543] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e731268, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:50:03,778-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3de7bc87] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:50:03,779-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3de7bc87] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:54:42,881-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [54e52a9a] Attempting to update VMs/Templates Ovf. >2018-05-29 05:54:42,884-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 05:54:42,884-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 05:54:42,885-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:54:42,889-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 05:54:42,892-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 05:54:42,892-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 05:54:42,892-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 05:54:42,892-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 05:54:42,893-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 05:54:42,893-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [60fa3968] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 05:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1334ce39] Autorecovering 2 hosts >2018-05-29 05:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1334ce39] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1a43e11d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25804489 >2018-05-29 05:55:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1a43e11d] FINISH, SetVdsStatusVDSCommand, log id: 25804489 >2018-05-29 05:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1a43e11d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 05:55:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1a43e11d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:55:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 05:55:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [697e2e66] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30d77e22 >2018-05-29 05:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [697e2e66] FINISH, SetVdsStatusVDSCommand, log id: 30d77e22 >2018-05-29 05:55:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [697e2e66] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71429938 >2018-05-29 05:55:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [697e2e66] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71429938 >2018-05-29 05:55:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 05:55:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [697e2e66] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 05:55:01,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2e6ba596] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5d80e857 >2018-05-29 05:55:01,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2e6ba596] FINISH, GetHardwareInfoVDSCommand, log id: 5d80e857 >2018-05-29 05:55:01,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3f449f2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 68568fb5 >2018-05-29 05:55:01,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3f449f2d] FINISH, GetHardwareInfoVDSCommand, log id: 68568fb5 >2018-05-29 05:55:01,342-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [13acbf89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:55:01,346-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [22049bcc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:55:01,351-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [50492d31] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:55:01,356-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1d084fbe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:55:01,357-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [70fe7a31] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:55:01,362-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [f03cae6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 05:55:01,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [70fe7a31] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16697e99 >2018-05-29 05:55:01,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [f03cae6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39a8d136 >2018-05-29 05:55:01,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:55:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [70fe7a31] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16697e99 >2018-05-29 05:55:01,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:01,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:01,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:01,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:01,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:01,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [70fe7a31] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:01,764-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [70fe7a31] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:55:01,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [70fe7a31] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70fe7a31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:55:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63264b5b >2018-05-29 05:55:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:55:01,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:55:01,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:55:01,872-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 05:55:01,872-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:55:01,872-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63264b5b >2018-05-29 05:55:01,872-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:55:01,885-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7597c16c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:55:01,886-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7597c16c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7913379a >2018-05-29 05:55:01,888-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7597c16c] FINISH, SetVdsStatusVDSCommand, log id: 7913379a >2018-05-29 05:55:01,921-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7597c16c, Job ID: b847008d-a016-494d-9b02-e41b4c690a14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:55:01,926-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50492d31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:55:01,928-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7597c16c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50492d31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:55:01,940-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5ccf1578] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 05:55:01,941-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5ccf1578] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 05:55:02,743-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 05:55:02,743-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [f03cae6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39a8d136 >2018-05-29 05:55:02,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:02,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:02,751-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:02,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:02,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 05:55:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [f03cae6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 05:55:02,761-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [f03cae6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 05:55:02,763-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [f03cae6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: f03cae6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 05:55:02,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 603c2b3e >2018-05-29 05:55:02,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:55:02,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 05:55:02,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 05:55:02,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 603c2b3e >2018-05-29 05:55:02,869-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 05:55:02,882-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [596ceec3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:55:02,883-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [596ceec3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e508659 >2018-05-29 05:55:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [596ceec3] FINISH, SetVdsStatusVDSCommand, log id: 4e508659 >2018-05-29 05:55:02,891-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 596ceec3, Job ID: 70f26230-4c1c-4fc5-bc8e-b59a9af84850, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 05:55:02,896-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d084fbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 05:55:02,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [596ceec3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d084fbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 05:55:02,909-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5c4014be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 05:55:02,910-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5c4014be] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [54215fb9] Autorecovering 2 hosts >2018-05-29 06:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [54215fb9] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5fdfff15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4024b344 >2018-05-29 06:00:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5fdfff15] FINISH, SetVdsStatusVDSCommand, log id: 4024b344 >2018-05-29 06:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5fdfff15] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5fdfff15] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:00:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:00:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [48c5774f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65c7d13c >2018-05-29 06:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [48c5774f] FINISH, SetVdsStatusVDSCommand, log id: 65c7d13c >2018-05-29 06:00:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [48c5774f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f7171dd >2018-05-29 06:00:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [48c5774f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f7171dd >2018-05-29 06:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:00:00,106-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [48c5774f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:00:00,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276a4e16] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f3756ed >2018-05-29 06:00:00,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276a4e16] FINISH, GetHardwareInfoVDSCommand, log id: 7f3756ed >2018-05-29 06:00:00,501-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2099bee7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:00:00,510-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2d3d3edb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:00:00,515-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [1b0b0897] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:00:00,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1b0b0897] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5bb00b57 >2018-05-29 06:00:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:00:00,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1b0b0897] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5bb00b57 >2018-05-29 06:00:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:00,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [1b0b0897] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:00,920-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [1b0b0897] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:00:00,923-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1b0b0897] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1b0b0897, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:00:00,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2838ea2f >2018-05-29 06:00:00,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:00:01,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:00:01,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:00:01,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2838ea2f >2018-05-29 06:00:01,028-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:00:01,042-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [22548bb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:00:01,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [22548bb4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65afb6a5 >2018-05-29 06:00:01,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [22548bb4] FINISH, SetVdsStatusVDSCommand, log id: 65afb6a5 >2018-05-29 06:00:01,079-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22548bb4, Job ID: 49a0ff3c-5478-4aef-9ba4-82a00630ee9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:00:01,084-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d3d3edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:00:01,086-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [22548bb4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d3d3edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:00:01,096-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [790e83c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:00:01,097-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [790e83c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:00:02,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [11ea5ef3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c7298de >2018-05-29 06:00:02,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [11ea5ef3] FINISH, GetHardwareInfoVDSCommand, log id: c7298de >2018-05-29 06:00:02,486-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [17aea6b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:00:02,490-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5dab5c7e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:00:02,496-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [43fa9c29] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:00:02,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [43fa9c29] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f919138 >2018-05-29 06:00:02,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:00:02,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [43fa9c29] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f919138 >2018-05-29 06:00:02,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:02,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:02,896-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:02,897-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:02,901-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:00:02,902-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [43fa9c29] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:00:02,904-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [43fa9c29] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:00:02,907-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43fa9c29] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43fa9c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:00:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 69040d28 >2018-05-29 06:00:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:00:03,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:00:03,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:00:03,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 69040d28 >2018-05-29 06:00:03,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:00:03,028-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [18dd31c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:00:03,029-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18dd31c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 497be59f >2018-05-29 06:00:03,031-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18dd31c2] FINISH, SetVdsStatusVDSCommand, log id: 497be59f >2018-05-29 06:00:03,064-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18dd31c2, Job ID: cc260127-3980-4fa6-b7b3-48e2fe7e53f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:00:03,069-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dab5c7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:00:03,071-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18dd31c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dab5c7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:00:03,082-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [673544b4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:00:03,083-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [673544b4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:05:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4802a040] Autorecovering 2 hosts >2018-05-29 06:05:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4802a040] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:05:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:05:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:05:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [556c3c99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 470041ac >2018-05-29 06:05:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [556c3c99] FINISH, SetVdsStatusVDSCommand, log id: 470041ac >2018-05-29 06:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [556c3c99] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:05:00,060-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [556c3c99] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [486ec98e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7035db28 >2018-05-29 06:05:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [486ec98e] FINISH, SetVdsStatusVDSCommand, log id: 7035db28 >2018-05-29 06:05:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [486ec98e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1dfe717c >2018-05-29 06:05:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [486ec98e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1dfe717c >2018-05-29 06:05:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:05:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [486ec98e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:05:00,667-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [85bfd40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 27f0d21c >2018-05-29 06:05:00,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [85bfd40] FINISH, GetHardwareInfoVDSCommand, log id: 27f0d21c >2018-05-29 06:05:00,685-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [343016d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:05:00,694-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6ee40a41] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:05:00,700-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [39abb8e1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:05:00,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [39abb8e1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ac4973c >2018-05-29 06:05:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:05:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [39abb8e1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ac4973c >2018-05-29 06:05:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:01,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:01,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:01,105-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:01,109-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:01,110-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [39abb8e1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:01,112-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [39abb8e1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:05:01,115-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [39abb8e1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39abb8e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:05:01,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 662c3b49 >2018-05-29 06:05:01,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:05:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:05:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:05:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 662c3b49 >2018-05-29 06:05:01,217-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:05:01,231-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [33e69171] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:05:01,232-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33e69171] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f19d695 >2018-05-29 06:05:01,234-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33e69171] FINISH, SetVdsStatusVDSCommand, log id: 3f19d695 >2018-05-29 06:05:01,266-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33e69171, Job ID: bf99815e-41c2-4582-8aa8-95015606dce9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:05:01,271-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ee40a41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:05:01,273-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33e69171] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ee40a41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:05:01,283-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3245e4d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:05:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3245e4d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:05:01,770-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [11158cee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73a341b4 >2018-05-29 06:05:01,772-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [11158cee] FINISH, GetHardwareInfoVDSCommand, log id: 73a341b4 >2018-05-29 06:05:01,785-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [635363e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:05:01,789-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [604bca53] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:05:01,795-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [496a6752] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:05:01,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [496a6752] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b4d1a04 >2018-05-29 06:05:02,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:05:02,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [496a6752] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b4d1a04 >2018-05-29 06:05:02,177-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:02,177-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:02,181-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:02,182-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:02,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:05:02,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [496a6752] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:05:02,190-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [496a6752] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:05:02,193-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [496a6752] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 496a6752, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:05:02,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44339707 >2018-05-29 06:05:02,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:05:02,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:05:02,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:05:02,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44339707 >2018-05-29 06:05:02,298-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:05:02,313-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [69caf525] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:05:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [69caf525] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a08b43c >2018-05-29 06:05:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [69caf525] FINISH, SetVdsStatusVDSCommand, log id: 5a08b43c >2018-05-29 06:05:02,350-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69caf525, Job ID: 703a980c-74f2-4da5-b63b-c201eb48e6f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:05:02,355-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 604bca53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:05:02,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69caf525] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 604bca53, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:05:02,374-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [42bdccd2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:05:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [42bdccd2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1adc2354] Autorecovering 2 hosts >2018-05-29 06:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1adc2354] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:10:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22e72255] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 793252ad >2018-05-29 06:10:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22e72255] FINISH, SetVdsStatusVDSCommand, log id: 793252ad >2018-05-29 06:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [22e72255] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:10:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [22e72255] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:10:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [14189bca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a4fa78d >2018-05-29 06:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [14189bca] FINISH, SetVdsStatusVDSCommand, log id: 5a4fa78d >2018-05-29 06:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [14189bca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 16d58975 >2018-05-29 06:10:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [14189bca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 16d58975 >2018-05-29 06:10:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:10:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [14189bca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:10:01,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [423aba30] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44eb1136 >2018-05-29 06:10:01,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [423aba30] FINISH, GetHardwareInfoVDSCommand, log id: 44eb1136 >2018-05-29 06:10:01,913-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [52f8dee6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:10:01,920-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [313209b7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:10:01,926-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [54d972b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:10:01,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [54d972b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ad987f3 >2018-05-29 06:10:02,315-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:10:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [54d972b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ad987f3 >2018-05-29 06:10:02,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:02,320-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:02,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:02,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:02,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:02,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [54d972b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:02,332-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [54d972b0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:10:02,335-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [54d972b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d972b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:10:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7103cc46 >2018-05-29 06:10:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:10:02,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:10:02,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:10:02,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7103cc46 >2018-05-29 06:10:02,438-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:10:02,452-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [221403fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:10:02,453-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [221403fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7eb99c08 >2018-05-29 06:10:02,455-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [221403fb] FINISH, SetVdsStatusVDSCommand, log id: 7eb99c08 >2018-05-29 06:10:02,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 221403fb, Job ID: 0bf7d187-5f70-458a-b701-3c8e9d5dd577, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:10:02,491-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 313209b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:10:02,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [221403fb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 313209b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:10:02,504-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7d88a9ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:10:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7d88a9ae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:10:03,017-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [e2d209a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6bd5629d >2018-05-29 06:10:03,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [e2d209a] FINISH, GetHardwareInfoVDSCommand, log id: 6bd5629d >2018-05-29 06:10:03,032-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [71faf09a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:10:03,040-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [1c71bf2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:10:03,045-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5362a0eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:10:03,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5362a0eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54057e63 >2018-05-29 06:10:03,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:10:03,420-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5362a0eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54057e63 >2018-05-29 06:10:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:03,424-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:03,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:03,429-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:03,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:10:03,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5362a0eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:10:03,436-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5362a0eb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:10:03,439-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5362a0eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5362a0eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:10:03,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6eac8288 >2018-05-29 06:10:03,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:10:03,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:10:03,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:10:03,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6eac8288 >2018-05-29 06:10:03,540-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:10:03,553-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [f4f3366] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:10:03,554-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [f4f3366] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c7d2494 >2018-05-29 06:10:03,556-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [f4f3366] FINISH, SetVdsStatusVDSCommand, log id: 1c7d2494 >2018-05-29 06:10:03,562-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f4f3366, Job ID: 666d6af8-1bb8-46fd-9396-2bb85415c28c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:10:03,566-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c71bf2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:10:03,568-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [f4f3366] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c71bf2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:10:03,579-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [57c6b1a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:10:03,580-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [57c6b1a6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:15:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3ea74022] Autorecovering 2 hosts >2018-05-29 06:15:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3ea74022] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bb94b13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11eacc22 >2018-05-29 06:15:00,083-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bb94b13] FINISH, SetVdsStatusVDSCommand, log id: 11eacc22 >2018-05-29 06:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5bb94b13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:15:00,086-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5bb94b13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:15:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f0122f4 >2018-05-29 06:15:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] FINISH, SetVdsStatusVDSCommand, log id: f0122f4 >2018-05-29 06:15:00,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2b574e7d >2018-05-29 06:15:00,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [a68e8c5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2b574e7d >2018-05-29 06:15:00,107-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:15:00,108-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [a68e8c5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:15:01,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6d7213d5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 16a6d41b >2018-05-29 06:15:01,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6d7213d5] FINISH, GetHardwareInfoVDSCommand, log id: 16a6d41b >2018-05-29 06:15:01,263-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [aa074bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:15:01,271-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [55369981] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:15:01,277-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [386708fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:15:01,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [386708fe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: c225796 >2018-05-29 06:15:02,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:15:02,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [386708fe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: c225796 >2018-05-29 06:15:02,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:02,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:02,674-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [386708fe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:02,682-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [386708fe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:15:02,685-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [386708fe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 386708fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:15:02,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 24ec33aa >2018-05-29 06:15:02,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:15:02,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:15:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:15:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:15:02,788-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:15:02,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 24ec33aa >2018-05-29 06:15:02,788-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:15:02,800-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [188715df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:15:02,802-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [188715df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 205aec82 >2018-05-29 06:15:02,804-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [188715df] FINISH, SetVdsStatusVDSCommand, log id: 205aec82 >2018-05-29 06:15:02,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 188715df, Job ID: 9321f4d1-5152-44d8-a06f-fea5bd9dca56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:15:02,840-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55369981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:15:02,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [188715df] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55369981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:15:02,853-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [aca1384] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:15:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [aca1384] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:15:03,129-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4e18a92f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a5ae3d6 >2018-05-29 06:15:03,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4e18a92f] FINISH, GetHardwareInfoVDSCommand, log id: 6a5ae3d6 >2018-05-29 06:15:03,143-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [461c26a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:15:03,151-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4afcfe76] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:15:03,156-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [200faa58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:15:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [200faa58] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f5f6121 >2018-05-29 06:15:03,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:15:03,544-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [200faa58] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f5f6121 >2018-05-29 06:15:03,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:03,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:03,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:03,553-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:03,557-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:15:03,558-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [200faa58] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:15:03,560-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [200faa58] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:15:03,563-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [200faa58] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 200faa58, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:15:03,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3f420dc6 >2018-05-29 06:15:03,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:15:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:15:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:15:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3f420dc6 >2018-05-29 06:15:03,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:15:03,677-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6925b705] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:15:03,678-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6925b705] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77c3ffc8 >2018-05-29 06:15:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6925b705] FINISH, SetVdsStatusVDSCommand, log id: 77c3ffc8 >2018-05-29 06:15:03,685-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6925b705, Job ID: c7303b68-5a57-4460-a68a-f656411a166c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:15:03,690-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4afcfe76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:15:03,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6925b705] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4afcfe76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:15:03,703-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [73ba7d1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:15:03,705-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [73ba7d1e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5e95e9de] Autorecovering 2 hosts >2018-05-29 06:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [5e95e9de] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:20:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [66f0e607] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 482b1666 >2018-05-29 06:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [66f0e607] FINISH, SetVdsStatusVDSCommand, log id: 482b1666 >2018-05-29 06:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [66f0e607] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:20:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [66f0e607] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:20:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d684cb3 >2018-05-29 06:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] FINISH, SetVdsStatusVDSCommand, log id: 4d684cb3 >2018-05-29 06:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3630c119 >2018-05-29 06:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [35ca54e6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3630c119 >2018-05-29 06:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:20:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [35ca54e6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:20:00,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2e5603a6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4893ab17 >2018-05-29 06:20:00,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2e5603a6] FINISH, GetHardwareInfoVDSCommand, log id: 4893ab17 >2018-05-29 06:20:00,569-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3f85bbb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:20:00,577-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4350d401] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:20:00,582-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [39687d9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:20:00,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [39687d9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22541ee2 >2018-05-29 06:20:00,967-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:20:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [39687d9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22541ee2 >2018-05-29 06:20:00,971-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:00,972-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:00,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:00,977-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:00,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:00,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [39687d9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:00,984-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [39687d9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:20:00,987-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39687d9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39687d9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:20:00,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9c13824 >2018-05-29 06:20:00,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:20:01,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:20:01,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:20:01,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9c13824 >2018-05-29 06:20:01,091-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:20:01,105-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [ecf004c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:20:01,107-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ecf004c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 832652b >2018-05-29 06:20:01,109-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ecf004c] FINISH, SetVdsStatusVDSCommand, log id: 832652b >2018-05-29 06:20:01,141-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ecf004c, Job ID: 092cce54-4018-433e-8d82-0de9a65ab100, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:20:01,146-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4350d401, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:20:01,149-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ecf004c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4350d401, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:20:01,160-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [313925f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:20:01,161-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [313925f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:20:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5475308b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 22eca576 >2018-05-29 06:20:01,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5475308b] FINISH, GetHardwareInfoVDSCommand, log id: 22eca576 >2018-05-29 06:20:01,300-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [35b8d1f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:20:01,309-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [175cf15] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:20:01,314-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30af2c43] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:20:01,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30af2c43] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f987cfb >2018-05-29 06:20:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:20:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30af2c43] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f987cfb >2018-05-29 06:20:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:01,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:01,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:01,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:20:01,718-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30af2c43] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:20:01,721-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30af2c43] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:20:01,724-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30af2c43] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30af2c43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:20:01,730-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a5ba819 >2018-05-29 06:20:01,730-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:20:01,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:20:01,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:20:01,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a5ba819 >2018-05-29 06:20:01,828-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:20:01,842-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [384f63f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:20:01,843-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [384f63f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75eb233a >2018-05-29 06:20:01,846-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [384f63f0] FINISH, SetVdsStatusVDSCommand, log id: 75eb233a >2018-05-29 06:20:01,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 384f63f0, Job ID: 31efce1a-0bc4-4af7-8bf5-efb63d80c390, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:20:01,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 175cf15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:20:01,884-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [384f63f0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 175cf15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:20:01,895-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [31cc5978] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:20:01,896-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [31cc5978] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:25:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [534c1ee6] Autorecovering 2 hosts >2018-05-29 06:25:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [534c1ee6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:25:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:25:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:25:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ac45e5d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3793a355 >2018-05-29 06:25:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ac45e5d] FINISH, SetVdsStatusVDSCommand, log id: 3793a355 >2018-05-29 06:25:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3ac45e5d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:25:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3ac45e5d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:25:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:25:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:25:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [43c5efb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20ebc2d1 >2018-05-29 06:25:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [43c5efb] FINISH, SetVdsStatusVDSCommand, log id: 20ebc2d1 >2018-05-29 06:25:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [43c5efb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a347812 >2018-05-29 06:25:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [43c5efb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a347812 >2018-05-29 06:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [43c5efb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:25:01,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [10669cef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5965d059 >2018-05-29 06:25:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [10669cef] FINISH, GetHardwareInfoVDSCommand, log id: 5965d059 >2018-05-29 06:25:01,840-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [18ca4696] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:25:01,844-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [18e4b306] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:25:01,849-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [59099b72] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:25:01,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [59099b72] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61243277 >2018-05-29 06:25:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [325068d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 455dea02 >2018-05-29 06:25:02,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [325068d3] FINISH, GetHardwareInfoVDSCommand, log id: 455dea02 >2018-05-29 06:25:02,598-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1d37f250] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:25:02,606-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [65defd17] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:25:02,612-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c755c5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:25:02,667-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c755c5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fe0916 >2018-05-29 06:25:03,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:25:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c755c5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fe0916 >2018-05-29 06:25:03,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c755c5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,064-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c755c5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:25:03,067-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c755c5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c755c5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:25:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 210c9523 >2018-05-29 06:25:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:25:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:25:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:25:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 210c9523 >2018-05-29 06:25:03,171-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:25:03,184-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1daa79ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:25:03,186-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1daa79ff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 409fea01 >2018-05-29 06:25:03,188-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1daa79ff] FINISH, SetVdsStatusVDSCommand, log id: 409fea01 >2018-05-29 06:25:03,221-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1daa79ff, Job ID: 68882bb5-3369-47f6-b6cc-3269be0df60c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:25:03,226-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65defd17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:25:03,228-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1daa79ff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65defd17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:25:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:25:03,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [59099b72] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61243277 >2018-05-29 06:25:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,240-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [47f0ae07] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:25:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,241-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [47f0ae07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:25:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:25:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [59099b72] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:25:03,248-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [59099b72] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:25:03,251-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [59099b72] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 59099b72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:25:03,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c326be5 >2018-05-29 06:25:03,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:25:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:25:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:25:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c326be5 >2018-05-29 06:25:03,354-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:25:03,368-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1a11e440] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:25:03,369-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1a11e440] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 108de41 >2018-05-29 06:25:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1a11e440] FINISH, SetVdsStatusVDSCommand, log id: 108de41 >2018-05-29 06:25:03,380-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1a11e440, Job ID: cfee0ab8-cf8f-4746-a8d3-35cf66068b6b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:25:03,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18e4b306, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:25:03,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1a11e440] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18e4b306, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:25:03,397-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [7d7bfd00] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:25:03,398-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7d7bfd00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [663e60b6] Autorecovering 2 hosts >2018-05-29 06:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [663e60b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d890adc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17640c7 >2018-05-29 06:30:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1d890adc] FINISH, SetVdsStatusVDSCommand, log id: 17640c7 >2018-05-29 06:30:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1d890adc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:30:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1d890adc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:30:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:30:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e739db3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fc78c9 >2018-05-29 06:30:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7e739db3] FINISH, SetVdsStatusVDSCommand, log id: 6fc78c9 >2018-05-29 06:30:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7e739db3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 48766094 >2018-05-29 06:30:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7e739db3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 48766094 >2018-05-29 06:30:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:30:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7e739db3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:30:00,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [75538bbf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: ba7758f >2018-05-29 06:30:00,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [75538bbf] FINISH, GetHardwareInfoVDSCommand, log id: ba7758f >2018-05-29 06:30:00,953-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6abc3387] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:30:00,962-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [682dc8a0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:30:00,967-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2a814799] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:30:01,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2a814799] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f8e38d0 >2018-05-29 06:30:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [692e6853] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2cb741e4 >2018-05-29 06:30:01,033-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [692e6853] FINISH, GetHardwareInfoVDSCommand, log id: 2cb741e4 >2018-05-29 06:30:01,047-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [27bc83c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:30:01,055-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6544d668] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:30:01,060-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [647096d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:30:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [647096d6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1da61315 >2018-05-29 06:30:01,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:30:01,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2a814799] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f8e38d0 >2018-05-29 06:30:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2a814799] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,423-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2a814799] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:30:01,426-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2a814799] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a814799, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:30:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d75b5d2 >2018-05-29 06:30:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:30:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:30:01,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [647096d6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1da61315 >2018-05-29 06:30:01,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,449-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:30:01,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [647096d6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:30:01,456-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [647096d6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:30:01,458-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647096d6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 647096d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:30:01,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:30:01,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:30:01,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d75b5d2 >2018-05-29 06:30:01,533-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:30:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11b564c8 >2018-05-29 06:30:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:30:01,547-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [141c773d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:30:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [141c773d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46d5b247 >2018-05-29 06:30:01,551-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [141c773d] FINISH, SetVdsStatusVDSCommand, log id: 46d5b247 >2018-05-29 06:30:01,583-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 141c773d, Job ID: cd5ba4ee-2c26-43f2-b45a-80460d40e574, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:30:01,587-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 682dc8a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:30:01,590-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [141c773d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 682dc8a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:30:01,600-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6dcf6495] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:30:01,601-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6dcf6495] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:30:01,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:30:01,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:30:01,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11b564c8 >2018-05-29 06:30:01,636-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:30:01,648-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [475357dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:30:01,650-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [475357dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38b07259 >2018-05-29 06:30:01,651-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [475357dc] FINISH, SetVdsStatusVDSCommand, log id: 38b07259 >2018-05-29 06:30:01,660-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 475357dc, Job ID: d49134a0-071d-4caf-bea3-9455a168b756, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:30:01,664-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6544d668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:30:01,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [475357dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6544d668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:30:01,676-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4d1ba88d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:30:01,678-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4d1ba88d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1ed9314d] Autorecovering 2 hosts >2018-05-29 06:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1ed9314d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:35:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4789fd9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f6085a4 >2018-05-29 06:35:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4789fd9] FINISH, SetVdsStatusVDSCommand, log id: 5f6085a4 >2018-05-29 06:35:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4789fd9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:35:00,049-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4789fd9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:35:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [601d822b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58cbe2c0 >2018-05-29 06:35:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [601d822b] FINISH, SetVdsStatusVDSCommand, log id: 58cbe2c0 >2018-05-29 06:35:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [601d822b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2f256356 >2018-05-29 06:35:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [601d822b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2f256356 >2018-05-29 06:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [601d822b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:35:02,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7cd4ff8d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 22fc88f4 >2018-05-29 06:35:02,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7cd4ff8d] FINISH, GetHardwareInfoVDSCommand, log id: 22fc88f4 >2018-05-29 06:35:02,384-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6cb6b724] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:35:02,388-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [399f2f32] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:35:02,393-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [c39f210] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:35:02,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [c39f210] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 267e9ecd >2018-05-29 06:35:02,570-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25c8bfe0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2439facb >2018-05-29 06:35:02,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25c8bfe0] FINISH, GetHardwareInfoVDSCommand, log id: 2439facb >2018-05-29 06:35:02,585-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1c7752c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:35:02,594-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2d66d31d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:35:02,599-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [78de10ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:35:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [78de10ae] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fe9cebd >2018-05-29 06:35:02,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:35:02,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [c39f210] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 267e9ecd >2018-05-29 06:35:02,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,791-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [c39f210] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,798-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [c39f210] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:35:02,800-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [c39f210] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c39f210, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:35:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37d00051 >2018-05-29 06:35:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:35:02,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:35:02,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:35:02,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37d00051 >2018-05-29 06:35:02,907-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:35:02,920-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [25abda3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:35:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [25abda3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c7a79d4 >2018-05-29 06:35:02,923-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [25abda3] FINISH, SetVdsStatusVDSCommand, log id: 7c7a79d4 >2018-05-29 06:35:02,956-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 25abda3, Job ID: c8c3ce74-f088-4795-9867-085d7ce240d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:35:02,960-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 399f2f32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:35:02,962-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [25abda3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 399f2f32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:35:02,972-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1f23ab24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:35:02,974-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1f23ab24] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:35:02,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:35:02,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [78de10ae] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fe9cebd >2018-05-29 06:35:02,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,981-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:35:02,989-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [78de10ae] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:35:02,993-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [78de10ae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:35:02,995-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78de10ae] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78de10ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:35:03,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7966d5c9 >2018-05-29 06:35:03,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:35:03,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:35:03,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:35:03,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7966d5c9 >2018-05-29 06:35:03,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:35:03,111-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [378f75de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:35:03,113-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [378f75de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cff1ee8 >2018-05-29 06:35:03,115-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [378f75de] FINISH, SetVdsStatusVDSCommand, log id: 4cff1ee8 >2018-05-29 06:35:03,143-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 378f75de, Job ID: 6f531223-897f-49e3-958a-b7cbeef26182, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:35:03,148-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d66d31d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:35:03,151-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [378f75de] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d66d31d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:35:03,161-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [ee4525a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:35:03,162-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [ee4525a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6dba9def] Autorecovering 2 hosts >2018-05-29 06:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6dba9def] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5bd50d79] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36fea794 >2018-05-29 06:40:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5bd50d79] FINISH, SetVdsStatusVDSCommand, log id: 36fea794 >2018-05-29 06:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5bd50d79] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:40:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5bd50d79] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [695167f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f47aa5f >2018-05-29 06:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [695167f9] FINISH, SetVdsStatusVDSCommand, log id: 3f47aa5f >2018-05-29 06:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [695167f9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5727c42b >2018-05-29 06:40:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [695167f9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5727c42b >2018-05-29 06:40:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:40:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [695167f9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:40:00,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1c85aa4f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62c44616 >2018-05-29 06:40:00,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1c85aa4f] FINISH, GetHardwareInfoVDSCommand, log id: 62c44616 >2018-05-29 06:40:00,605-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ee27b1d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:40:00,613-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [e36636d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:40:00,618-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3a37b3c0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:40:00,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3a37b3c0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 67141f80 >2018-05-29 06:40:00,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [39c925f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 788bade2 >2018-05-29 06:40:00,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [39c925f9] FINISH, GetHardwareInfoVDSCommand, log id: 788bade2 >2018-05-29 06:40:00,808-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1ffa3ca8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:40:00,812-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3d774928] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:40:00,817-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55f6d92e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:40:00,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55f6d92e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3891cdf6 >2018-05-29 06:40:01,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:40:01,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3a37b3c0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 67141f80 >2018-05-29 06:40:01,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,016-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,020-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,021-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3a37b3c0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,029-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3a37b3c0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:40:01,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3a37b3c0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a37b3c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:40:01,038-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2beb5338 >2018-05-29 06:40:01,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:40:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:40:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:40:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2beb5338 >2018-05-29 06:40:01,135-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:40:01,148-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [610611ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:40:01,149-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [610611ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2226ed5d >2018-05-29 06:40:01,151-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [610611ca] FINISH, SetVdsStatusVDSCommand, log id: 2226ed5d >2018-05-29 06:40:01,183-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 610611ca, Job ID: 1a679d65-4457-489b-8ecb-bd39b8bb59e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:40:01,187-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e36636d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:40:01,189-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [610611ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e36636d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:40:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:40:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [55f6d92e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3891cdf6 >2018-05-29 06:40:01,201-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [10853814] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:40:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [10853814] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:40:01,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,208-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:40:01,212-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [55f6d92e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:40:01,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [55f6d92e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:40:01,218-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [55f6d92e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55f6d92e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:40:01,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 250b76c5 >2018-05-29 06:40:01,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:40:01,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:40:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:40:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:40:01,324-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:40:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 250b76c5 >2018-05-29 06:40:01,325-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:40:01,337-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [14a76402] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:40:01,338-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [14a76402] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b4f90db >2018-05-29 06:40:01,341-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [14a76402] FINISH, SetVdsStatusVDSCommand, log id: 6b4f90db >2018-05-29 06:40:01,349-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 14a76402, Job ID: 75212b5b-40b2-4e5d-9ef9-cae8b439b8d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:40:01,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d774928, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:40:01,356-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14a76402] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d774928, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:40:01,366-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [13b6194] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:40:01,367-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [13b6194] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60370174] Autorecovering 2 hosts >2018-05-29 06:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60370174] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:45:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [625b28c9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fb347bc >2018-05-29 06:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [625b28c9] FINISH, SetVdsStatusVDSCommand, log id: 7fb347bc >2018-05-29 06:45:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [625b28c9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:45:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [625b28c9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:45:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 200ebc51 >2018-05-29 06:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] FINISH, SetVdsStatusVDSCommand, log id: 200ebc51 >2018-05-29 06:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1a2eba86 >2018-05-29 06:45:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [1905e3fd] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1a2eba86 >2018-05-29 06:45:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1905e3fd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:45:01,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4f0b9e5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c9600f4 >2018-05-29 06:45:01,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4f0b9e5d] FINISH, GetHardwareInfoVDSCommand, log id: c9600f4 >2018-05-29 06:45:01,886-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [69ffd17c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:45:01,894-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [65d6a0c3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:45:01,899-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eb06301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:45:01,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eb06301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 540ba70f >2018-05-29 06:45:02,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [37b5c1cd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 48d1e8ce >2018-05-29 06:45:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [37b5c1cd] FINISH, GetHardwareInfoVDSCommand, log id: 48d1e8ce >2018-05-29 06:45:02,259-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [272e8c7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:45:02,264-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [400918e3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:45:02,269-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d4dc7d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:45:02,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d4dc7d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fcba56e >2018-05-29 06:45:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:45:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eb06301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 540ba70f >2018-05-29 06:45:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:02,343-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:02,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:02,348-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:02,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:02,352-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eb06301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:02,354-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eb06301] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:45:02,357-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eb06301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5eb06301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:45:02,364-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ca808d3 >2018-05-29 06:45:02,364-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:45:02,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:45:02,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:45:02,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ca808d3 >2018-05-29 06:45:02,463-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:45:02,476-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5d8989ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:45:02,478-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5d8989ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290f833b >2018-05-29 06:45:02,480-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5d8989ed] FINISH, SetVdsStatusVDSCommand, log id: 290f833b >2018-05-29 06:45:02,514-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d8989ed, Job ID: ebaf20b4-1b02-4d93-98d1-1ced8ad4b32f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:45:02,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65d6a0c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:45:02,521-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5d8989ed] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65d6a0c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:45:02,532-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [28186840] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:45:02,533-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [28186840] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:45:04,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:45:04,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d4dc7d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fcba56e >2018-05-29 06:45:04,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:04,594-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:04,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:04,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:04,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:45:04,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d4dc7d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:45:04,607-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d4dc7d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:45:04,609-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d4dc7d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d4dc7d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:45:04,612-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e3f66f6 >2018-05-29 06:45:04,612-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:45:04,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:45:04,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:45:04,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e3f66f6 >2018-05-29 06:45:04,709-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:45:04,723-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5fda4dc3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:45:04,724-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5fda4dc3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43866b1 >2018-05-29 06:45:04,726-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5fda4dc3] FINISH, SetVdsStatusVDSCommand, log id: 43866b1 >2018-05-29 06:45:04,756-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5fda4dc3, Job ID: 1548856c-039e-494b-8706-96c4cf43ce89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:45:04,761-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 400918e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:45:04,763-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5fda4dc3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 400918e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:45:04,774-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [b8ee3aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:45:04,775-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [b8ee3aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [627e71c0] Autorecovering 2 hosts >2018-05-29 06:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [627e71c0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:50:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50a822ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7673280f >2018-05-29 06:50:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50a822ab] FINISH, SetVdsStatusVDSCommand, log id: 7673280f >2018-05-29 06:50:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50a822ab] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:50:00,051-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50a822ab] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:50:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52183274] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 666a08f6 >2018-05-29 06:50:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52183274] FINISH, SetVdsStatusVDSCommand, log id: 666a08f6 >2018-05-29 06:50:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [52183274] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79762ae4 >2018-05-29 06:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [52183274] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79762ae4 >2018-05-29 06:50:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52183274] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:50:02,455-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [46f097d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e2bffd3 >2018-05-29 06:50:02,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [46f097d4] FINISH, GetHardwareInfoVDSCommand, log id: 4e2bffd3 >2018-05-29 06:50:02,471-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [697fbceb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:50:02,475-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6292c433] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:50:02,480-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5b12e033] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:50:02,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5b12e033] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ebdfdb4 >2018-05-29 06:50:02,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:50:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5b12e033] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ebdfdb4 >2018-05-29 06:50:02,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:02,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:02,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:02,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:02,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:02,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5b12e033] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:02,926-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5b12e033] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:50:02,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5b12e033] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b12e033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:50:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48e7b12f >2018-05-29 06:50:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:50:03,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:50:03,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:50:03,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48e7b12f >2018-05-29 06:50:03,037-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:50:03,052-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7a2f618e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:50:03,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7a2f618e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7628fbaf >2018-05-29 06:50:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7a2f618e] FINISH, SetVdsStatusVDSCommand, log id: 7628fbaf >2018-05-29 06:50:03,087-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a2f618e, Job ID: 556cc801-306d-4ac6-bb77-151ceb469ba7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:50:03,092-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6292c433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:50:03,094-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a2f618e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6292c433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:50:03,105-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2aed90aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:50:03,106-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2aed90aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:50:03,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5349faef] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5a220e7a >2018-05-29 06:50:03,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5349faef] FINISH, GetHardwareInfoVDSCommand, log id: 5a220e7a >2018-05-29 06:50:03,156-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [46b7bfaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:50:03,165-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3ef776c5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:50:03,170-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2c890233] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:50:03,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2c890233] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 45a8e617 >2018-05-29 06:50:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:50:03,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2c890233] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 45a8e617 >2018-05-29 06:50:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:03,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:03,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:03,573-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:03,577-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:50:03,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2c890233] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:50:03,581-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2c890233] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:50:03,583-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2c890233] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c890233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:50:03,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c5fa7b >2018-05-29 06:50:03,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:50:03,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:50:03,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:50:03,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c5fa7b >2018-05-29 06:50:03,684-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:50:03,697-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [269befa6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:50:03,698-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [269befa6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17cf0b62 >2018-05-29 06:50:03,701-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [269befa6] FINISH, SetVdsStatusVDSCommand, log id: 17cf0b62 >2018-05-29 06:50:03,706-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 269befa6, Job ID: 0cf880b8-84b1-45dd-ad32-ba6841d563ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:50:03,711-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3ef776c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:50:03,713-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [269befa6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3ef776c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:50:03,724-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [79958cd6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:50:03,725-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [79958cd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:54:42,893-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [4478f03c] Attempting to update VMs/Templates Ovf. >2018-05-29 06:54:42,896-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 06:54:42,896-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 06:54:42,896-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:54:42,901-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 06:54:42,904-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 06:54:42,904-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 06:54:42,904-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 06:54:42,904-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 06:54:42,905-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 06:54:42,905-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [300ed517] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 06:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [70406a73] Autorecovering 2 hosts >2018-05-29 06:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [70406a73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:55:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [9acbffd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1354c90e >2018-05-29 06:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [9acbffd] FINISH, SetVdsStatusVDSCommand, log id: 1354c90e >2018-05-29 06:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [9acbffd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 06:55:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [9acbffd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 06:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [214409f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f291a09 >2018-05-29 06:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [214409f] FINISH, SetVdsStatusVDSCommand, log id: 6f291a09 >2018-05-29 06:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [214409f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 62d69250 >2018-05-29 06:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [214409f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 62d69250 >2018-05-29 06:55:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 06:55:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [214409f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 06:55:00,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67bce0e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 72f619d >2018-05-29 06:55:00,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [67bce0e7] FINISH, GetHardwareInfoVDSCommand, log id: 72f619d >2018-05-29 06:55:00,776-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [797ad905] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:55:00,784-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [177b6c9f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:55:00,789-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3afec7c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:55:00,800-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3afec7c2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 659d0c20 >2018-05-29 06:55:01,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:55:01,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3afec7c2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 659d0c20 >2018-05-29 06:55:01,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,188-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3afec7c2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,191-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3afec7c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:55:01,193-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3afec7c2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3afec7c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:55:01,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76561eca >2018-05-29 06:55:01,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:55:01,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:55:01,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:55:01,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76561eca >2018-05-29 06:55:01,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:55:01,310-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [494b55ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:55:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [494b55ee] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c75d74f >2018-05-29 06:55:01,313-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [494b55ee] FINISH, SetVdsStatusVDSCommand, log id: 6c75d74f >2018-05-29 06:55:01,345-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 494b55ee, Job ID: c8fd0afb-0c2b-41ba-b40f-e26617ba890d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:55:01,350-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 177b6c9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:55:01,352-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [494b55ee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 177b6c9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:55:01,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [78a111a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 555cb372 >2018-05-29 06:55:01,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [78a111a] FINISH, GetHardwareInfoVDSCommand, log id: 555cb372 >2018-05-29 06:55:01,364-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [201d2106] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 06:55:01,365-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [201d2106] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 06:55:01,396-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4c6827c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:55:01,404-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [528e722c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:55:01,410-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2658a34] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 06:55:01,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2658a34] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63cbb68a >2018-05-29 06:55:01,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 06:55:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2658a34] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63cbb68a >2018-05-29 06:55:01,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,852-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,856-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,857-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 06:55:01,861-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2658a34] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 06:55:01,864-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2658a34] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 06:55:01,867-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2658a34] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2658a34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 06:55:01,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 85b3b79 >2018-05-29 06:55:01,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:55:01,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 06:55:01,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 06:55:01,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 85b3b79 >2018-05-29 06:55:01,969-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 06:55:01,983-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [eee1357] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:55:01,985-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [eee1357] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 275d4bb6 >2018-05-29 06:55:01,987-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [eee1357] FINISH, SetVdsStatusVDSCommand, log id: 275d4bb6 >2018-05-29 06:55:02,019-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: eee1357, Job ID: 2fa79f6d-d0f2-4aaf-b9d8-a9d5a64c2c48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 06:55:02,023-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 528e722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 06:55:02,025-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [eee1357] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 528e722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 06:55:02,037-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [25aaff6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 06:55:02,038-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [25aaff6f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45ebefb1] Autorecovering 2 hosts >2018-05-29 07:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45ebefb1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55fb9889] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4958bfab >2018-05-29 07:00:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55fb9889] FINISH, SetVdsStatusVDSCommand, log id: 4958bfab >2018-05-29 07:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [55fb9889] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [55fb9889] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33841fb0 >2018-05-29 07:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] FINISH, SetVdsStatusVDSCommand, log id: 33841fb0 >2018-05-29 07:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 413fc040 >2018-05-29 07:00:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6cd2f160] FINISH, SetHaMaintenanceModeVDSCommand, log id: 413fc040 >2018-05-29 07:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:00:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6cd2f160] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:00:02,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [338a57ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44b0ec8c >2018-05-29 07:00:02,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [338a57ab] FINISH, GetHardwareInfoVDSCommand, log id: 44b0ec8c >2018-05-29 07:00:02,264-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [66e475ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:00:02,273-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2cd74724] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:00:02,278-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [19cab49e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:00:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [19cab49e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12b6e7a4 >2018-05-29 07:00:02,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:00:02,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [19cab49e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12b6e7a4 >2018-05-29 07:00:02,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:02,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:02,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:02,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [19cab49e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:02,668-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [19cab49e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:00:02,671-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19cab49e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 19cab49e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:00:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 142a3ea4 >2018-05-29 07:00:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:00:02,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dba5295] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26ab5baa >2018-05-29 07:00:02,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dba5295] FINISH, GetHardwareInfoVDSCommand, log id: 26ab5baa >2018-05-29 07:00:02,766-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [702929f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:00:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:00:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:00:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 142a3ea4 >2018-05-29 07:00:02,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:00:02,775-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [66e465b9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:00:02,781-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1af6c0a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:00:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1af6c0a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29cf702 >2018-05-29 07:00:02,788-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [75489e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:00:02,789-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [75489e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a6ddd7e >2018-05-29 07:00:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [75489e3] FINISH, SetVdsStatusVDSCommand, log id: 1a6ddd7e >2018-05-29 07:00:02,822-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 75489e3, Job ID: dffb8be6-5bfe-454b-aa44-ac82b4b3031c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:00:02,827-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2cd74724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:00:02,829-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [75489e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2cd74724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:00:02,840-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [30596759] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:00:02,841-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [30596759] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:00:03,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:00:03,165-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1af6c0a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29cf702 >2018-05-29 07:00:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:03,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:03,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:03,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:00:03,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1af6c0a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:00:03,182-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1af6c0a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:00:03,184-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1af6c0a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1af6c0a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:00:03,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13be14b8 >2018-05-29 07:00:03,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:00:03,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:00:03,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:00:03,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13be14b8 >2018-05-29 07:00:03,295-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:00:03,309-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [8b53a89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:00:03,310-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [8b53a89] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 389c9c4b >2018-05-29 07:00:03,312-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [8b53a89] FINISH, SetVdsStatusVDSCommand, log id: 389c9c4b >2018-05-29 07:00:03,318-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8b53a89, Job ID: 86d3b5fc-985e-43c9-9de6-a8631f359583, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:00:03,323-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66e465b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:00:03,325-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8b53a89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66e465b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:00:03,336-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [48a4f5d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:00:03,337-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [48a4f5d1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5c6287da] Autorecovering 2 hosts >2018-05-29 07:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5c6287da] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:05:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [73ea879e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37cfdcef >2018-05-29 07:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [73ea879e] FINISH, SetVdsStatusVDSCommand, log id: 37cfdcef >2018-05-29 07:05:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [73ea879e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [73ea879e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:05:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [892938a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b8bb679 >2018-05-29 07:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [892938a] FINISH, SetVdsStatusVDSCommand, log id: b8bb679 >2018-05-29 07:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [892938a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1bf66285 >2018-05-29 07:05:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [892938a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1bf66285 >2018-05-29 07:05:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [892938a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:05:00,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1ebbfb73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b2e7a61 >2018-05-29 07:05:00,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1ebbfb73] FINISH, GetHardwareInfoVDSCommand, log id: 4b2e7a61 >2018-05-29 07:05:00,501-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7a1754ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:05:00,509-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [53796d10] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:05:00,514-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [48bcb69b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:05:00,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [48bcb69b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a1a00e8 >2018-05-29 07:05:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:05:00,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [48bcb69b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3a1a00e8 >2018-05-29 07:05:00,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6cf6a443] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 56301026 >2018-05-29 07:05:00,911-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6cf6a443] FINISH, GetHardwareInfoVDSCommand, log id: 56301026 >2018-05-29 07:05:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:00,913-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:00,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:00,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:00,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [48bcb69b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:00,924-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1b69d1fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:05:00,926-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [48bcb69b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:05:00,928-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6306895a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:05:00,929-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [48bcb69b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 48bcb69b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:05:00,933-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [12a0d5cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:05:00,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e3e4b2 >2018-05-29 07:05:00,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:05:00,946-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [12a0d5cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2691e67a >2018-05-29 07:05:01,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:05:01,033-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:05:01,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:05:01,034-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:05:01,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e3e4b2 >2018-05-29 07:05:01,034-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:05:01,046-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2e95eadf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:05:01,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e95eadf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5da3dc68 >2018-05-29 07:05:01,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e95eadf] FINISH, SetVdsStatusVDSCommand, log id: 5da3dc68 >2018-05-29 07:05:01,057-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e95eadf, Job ID: 24c20fc8-7899-4cb5-8728-671ed2dd2355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:05:01,062-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 53796d10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:05:01,064-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2e95eadf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 53796d10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:05:01,075-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5ab6961e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:05:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5ab6961e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:05:01,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:05:01,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [12a0d5cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2691e67a >2018-05-29 07:05:01,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:01,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:01,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:01,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:01,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:05:01,345-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [12a0d5cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:05:01,349-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [12a0d5cf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:05:01,351-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12a0d5cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12a0d5cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:05:01,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a4496b4 >2018-05-29 07:05:01,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:05:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:05:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:05:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a4496b4 >2018-05-29 07:05:01,459-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:05:01,472-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2653bdc2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:05:01,474-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2653bdc2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 385261e5 >2018-05-29 07:05:01,476-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2653bdc2] FINISH, SetVdsStatusVDSCommand, log id: 385261e5 >2018-05-29 07:05:01,481-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2653bdc2, Job ID: 5a1e62ea-7f04-4f1d-a6c5-efca81f00b15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:05:01,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6306895a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:05:01,488-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2653bdc2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6306895a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:05:01,499-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [741fc064] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:05:01,500-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [741fc064] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:10:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5c07c8ef] Autorecovering 2 hosts >2018-05-29 07:10:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5c07c8ef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1bef4147] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 436fd524 >2018-05-29 07:10:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1bef4147] FINISH, SetVdsStatusVDSCommand, log id: 436fd524 >2018-05-29 07:10:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1bef4147] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:10:00,085-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1bef4147] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e326b19 >2018-05-29 07:10:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] FINISH, SetVdsStatusVDSCommand, log id: 4e326b19 >2018-05-29 07:10:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 72669b1b >2018-05-29 07:10:00,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5ec94dd1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 72669b1b >2018-05-29 07:10:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:10:00,106-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ec94dd1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:10:01,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [537c2ec0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 794af37f >2018-05-29 07:10:01,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [537c2ec0] FINISH, GetHardwareInfoVDSCommand, log id: 794af37f >2018-05-29 07:10:01,785-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7f633873] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:10:01,793-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [77f45fcc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:10:01,798-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [661333d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:10:01,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [661333d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17465723 >2018-05-29 07:10:02,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [277060d8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 770b6361 >2018-05-29 07:10:02,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [277060d8] FINISH, GetHardwareInfoVDSCommand, log id: 770b6361 >2018-05-29 07:10:02,112-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [60ad0554] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:10:02,116-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7b9675aa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:10:02,122-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22759953] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:10:02,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22759953] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ba54580 >2018-05-29 07:10:02,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:10:02,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [661333d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17465723 >2018-05-29 07:10:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,188-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [661333d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,193-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [661333d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:10:02,196-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [661333d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 661333d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:10:02,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bfdb66 >2018-05-29 07:10:02,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bfdb66 >2018-05-29 07:10:02,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:10:02,312-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6fdcf58d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:10:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6fdcf58d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2813ecd0 >2018-05-29 07:10:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6fdcf58d] FINISH, SetVdsStatusVDSCommand, log id: 2813ecd0 >2018-05-29 07:10:02,349-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fdcf58d, Job ID: 831af585-870b-4ce9-86ed-f766fb50a5bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:10:02,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77f45fcc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:10:02,357-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6fdcf58d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77f45fcc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:10:02,368-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [76188f8e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:10:02,369-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [76188f8e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:10:02,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:10:02,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22759953] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ba54580 >2018-05-29 07:10:02,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,519-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,520-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:10:02,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22759953] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:10:02,528-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22759953] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:10:02,531-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22759953] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 22759953, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:10:02,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a2c6f61 >2018-05-29 07:10:02,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:10:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:10:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:10:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a2c6f61 >2018-05-29 07:10:02,633-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:10:02,647-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42361695] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:10:02,648-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42361695] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 133854d0 >2018-05-29 07:10:02,650-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42361695] FINISH, SetVdsStatusVDSCommand, log id: 133854d0 >2018-05-29 07:10:02,656-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42361695, Job ID: 7620d78f-3668-4e9d-9e9e-7fb36ce7a80e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:10:02,660-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b9675aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:10:02,662-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42361695] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b9675aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:10:02,673-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [23e28e2f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:10:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [23e28e2f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4e8bb5fd] Autorecovering 2 hosts >2018-05-29 07:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4e8bb5fd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [616431c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8dcb6e8 >2018-05-29 07:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [616431c6] FINISH, SetVdsStatusVDSCommand, log id: 8dcb6e8 >2018-05-29 07:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [616431c6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:15:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [616431c6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:15:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60899ed4 >2018-05-29 07:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] FINISH, SetVdsStatusVDSCommand, log id: 60899ed4 >2018-05-29 07:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b0f7a7e >2018-05-29 07:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6a55c1ed] FINISH, SetHaMaintenanceModeVDSCommand, log id: b0f7a7e >2018-05-29 07:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a55c1ed] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:15:00,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [492c37c2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1af4147c >2018-05-29 07:15:00,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [492c37c2] FINISH, GetHardwareInfoVDSCommand, log id: 1af4147c >2018-05-29 07:15:00,282-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3f14d45f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:15:00,290-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [8488820] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:15:00,295-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1a6f6f8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:15:00,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1a6f6f8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: dc5c963 >2018-05-29 07:15:00,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:15:00,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1a6f6f8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: dc5c963 >2018-05-29 07:15:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:00,682-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:00,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:00,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:00,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1a6f6f8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:00,694-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1a6f6f8f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:15:00,696-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1a6f6f8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a6f6f8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:15:00,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75e60f8d >2018-05-29 07:15:00,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:15:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:15:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:15:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75e60f8d >2018-05-29 07:15:00,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:15:00,811-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6fdfada9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:15:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6fdfada9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cdf06f7 >2018-05-29 07:15:00,815-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6fdfada9] FINISH, SetVdsStatusVDSCommand, log id: cdf06f7 >2018-05-29 07:15:00,844-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fdfada9, Job ID: 473371bd-84f0-4eec-825a-ada20ee30c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:15:00,848-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8488820, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:15:00,851-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6fdfada9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8488820, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:15:00,862-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [316d39a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:15:00,863-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [316d39a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:15:03,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [49a52075] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 642a002a >2018-05-29 07:15:03,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [49a52075] FINISH, GetHardwareInfoVDSCommand, log id: 642a002a >2018-05-29 07:15:03,889-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [410abb6b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:15:03,898-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [33faf598] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:15:03,904-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4cd2a66e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:15:03,915-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4cd2a66e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e915584 >2018-05-29 07:15:05,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:15:05,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4cd2a66e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e915584 >2018-05-29 07:15:05,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:05,292-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:05,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:05,297-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:05,301-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:15:05,302-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4cd2a66e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:15:05,304-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4cd2a66e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:15:05,307-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4cd2a66e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4cd2a66e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:15:05,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73b0a00f >2018-05-29 07:15:05,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:15:05,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:15:05,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:15:05,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73b0a00f >2018-05-29 07:15:05,413-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:15:05,427-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [56c32cf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:15:05,429-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c32cf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c6d4760 >2018-05-29 07:15:05,431-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c32cf5] FINISH, SetVdsStatusVDSCommand, log id: c6d4760 >2018-05-29 07:15:05,464-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56c32cf5, Job ID: 6bd3e4ba-4b43-47ca-80cc-f92ab5d835ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:15:05,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33faf598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:15:05,471-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [56c32cf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33faf598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:15:05,483-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [64241e8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:15:05,484-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [64241e8f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57a01d45] Autorecovering 2 hosts >2018-05-29 07:20:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57a01d45] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [60204a7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b45b413 >2018-05-29 07:20:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [60204a7b] FINISH, SetVdsStatusVDSCommand, log id: 3b45b413 >2018-05-29 07:20:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [60204a7b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:20:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [60204a7b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,094-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler2) [308231df] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS_INIT]', sharedLocks=''}' >2018-05-29 07:20:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [308231df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17d8adff >2018-05-29 07:20:00,108-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [308231df] FINISH, SetVdsStatusVDSCommand, log id: 17d8adff >2018-05-29 07:20:00,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [308231df] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3480c8df >2018-05-29 07:20:00,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [308231df] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3480c8df >2018-05-29 07:20:00,113-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:20:00,114-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [308231df] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:20:01,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [18ac7b8d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 69909e6a >2018-05-29 07:20:01,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [18ac7b8d] FINISH, GetHardwareInfoVDSCommand, log id: 69909e6a >2018-05-29 07:20:01,535-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6b476574] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:20:01,546-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [29857e25] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:20:01,551-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6ad84f30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:20:01,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6ad84f30] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7981dcae >2018-05-29 07:20:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:20:01,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6ad84f30] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7981dcae >2018-05-29 07:20:01,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:01,952-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:01,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:01,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:01,961-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6ad84f30] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:01,965-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6ad84f30] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:20:01,968-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6ad84f30] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ad84f30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:20:01,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c32a35a >2018-05-29 07:20:01,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:20:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:20:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:20:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c32a35a >2018-05-29 07:20:02,075-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:20:02,090-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2a267dd5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:20:02,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2a267dd5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a40e21f >2018-05-29 07:20:02,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2a267dd5] FINISH, SetVdsStatusVDSCommand, log id: 7a40e21f >2018-05-29 07:20:02,125-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2a267dd5, Job ID: d50e3048-49e4-4f11-bbe0-fb1b80076a36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:20:02,130-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29857e25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:20:02,132-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2a267dd5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29857e25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:20:02,144-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [534659a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:20:02,145-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [534659a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:20:03,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c7d057a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 61f46cdf >2018-05-29 07:20:03,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [c7d057a] FINISH, GetHardwareInfoVDSCommand, log id: 61f46cdf >2018-05-29 07:20:03,250-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [38f074f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:20:03,258-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7787e12a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:20:03,263-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4825892f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:20:03,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4825892f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15a0460a >2018-05-29 07:20:03,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:20:03,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4825892f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15a0460a >2018-05-29 07:20:03,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:03,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:03,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:03,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:20:03,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4825892f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:20:03,662-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4825892f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:20:03,665-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4825892f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4825892f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:20:03,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5aa93546 >2018-05-29 07:20:03,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:20:03,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:20:03,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:20:03,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5aa93546 >2018-05-29 07:20:03,769-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:20:03,781-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [5d62a5d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:20:03,783-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5d62a5d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f942773 >2018-05-29 07:20:03,784-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5d62a5d3] FINISH, SetVdsStatusVDSCommand, log id: 7f942773 >2018-05-29 07:20:03,816-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d62a5d3, Job ID: 38a4bd4e-9080-47c9-876f-cc5c7b5ebd16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:20:03,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7787e12a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:20:03,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d62a5d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7787e12a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:20:03,833-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [49c3ae8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:20:03,834-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [49c3ae8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a95c25d] Autorecovering 2 hosts >2018-05-29 07:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7a95c25d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:25:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61caaae7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4086c52 >2018-05-29 07:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61caaae7] FINISH, SetVdsStatusVDSCommand, log id: 4086c52 >2018-05-29 07:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61caaae7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:25:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [61caaae7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a0662ef >2018-05-29 07:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] FINISH, SetVdsStatusVDSCommand, log id: 7a0662ef >2018-05-29 07:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73150d2b >2018-05-29 07:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7b92ae8f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73150d2b >2018-05-29 07:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:25:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7b92ae8f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:25:01,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [44c8fa9a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2ab307d9 >2018-05-29 07:25:01,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [44c8fa9a] FINISH, GetHardwareInfoVDSCommand, log id: 2ab307d9 >2018-05-29 07:25:01,651-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [52f5fcad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:25:01,660-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [66a24ed5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:25:01,665-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5d8c74da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:25:01,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5d8c74da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31e51a01 >2018-05-29 07:25:02,039-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:25:02,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5d8c74da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31e51a01 >2018-05-29 07:25:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:02,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:02,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:02,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:02,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5d8c74da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:02,055-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5d8c74da] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:25:02,058-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8c74da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d8c74da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:25:02,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f9d4ff8 >2018-05-29 07:25:02,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:25:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:25:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:25:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f9d4ff8 >2018-05-29 07:25:02,160-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:25:02,174-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [555d1925] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:25:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555d1925] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 374b066a >2018-05-29 07:25:02,177-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555d1925] FINISH, SetVdsStatusVDSCommand, log id: 374b066a >2018-05-29 07:25:02,207-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 555d1925, Job ID: c7995c02-020e-4a4d-bc6a-8645928617a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:25:02,212-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66a24ed5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:25:02,214-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555d1925] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66a24ed5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:25:02,225-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [64f4eaf0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:25:02,226-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [64f4eaf0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:25:02,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [71c7f937] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dfd8e8c >2018-05-29 07:25:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [71c7f937] FINISH, GetHardwareInfoVDSCommand, log id: 3dfd8e8c >2018-05-29 07:25:02,880-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4cd34317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:25:02,884-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6303f0d1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:25:02,890-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [442edd05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:25:02,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [442edd05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4210329f >2018-05-29 07:25:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:25:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [442edd05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4210329f >2018-05-29 07:25:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:03,284-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:03,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:25:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [442edd05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:25:03,296-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [442edd05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:25:03,299-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [442edd05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 442edd05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:25:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b18e531 >2018-05-29 07:25:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:25:03,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:25:03,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:25:03,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b18e531 >2018-05-29 07:25:03,402-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:25:03,415-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2c4f330] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:25:03,416-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2c4f330] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1195bf15 >2018-05-29 07:25:03,418-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2c4f330] FINISH, SetVdsStatusVDSCommand, log id: 1195bf15 >2018-05-29 07:25:03,453-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c4f330, Job ID: 390922b1-0e29-4ebc-a0f1-6186c790538a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:25:03,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6303f0d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:25:03,459-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2c4f330] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6303f0d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:25:03,470-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2232abd5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:25:03,471-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2232abd5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:30:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [354a3052] Autorecovering 2 hosts >2018-05-29 07:30:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [354a3052] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:30:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:30:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:30:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [13d3c5c5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4894dcd6 >2018-05-29 07:30:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [13d3c5c5] FINISH, SetVdsStatusVDSCommand, log id: 4894dcd6 >2018-05-29 07:30:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [13d3c5c5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:30:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [13d3c5c5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:30:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:30:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:30:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [689277f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70fee738 >2018-05-29 07:30:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [689277f6] FINISH, SetVdsStatusVDSCommand, log id: 70fee738 >2018-05-29 07:30:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [689277f6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a77c610 >2018-05-29 07:30:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [689277f6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a77c610 >2018-05-29 07:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [689277f6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:30:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [53cafe1e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b3924a9 >2018-05-29 07:30:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [53cafe1e] FINISH, GetHardwareInfoVDSCommand, log id: 2b3924a9 >2018-05-29 07:30:01,085-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [10a43b3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:30:01,090-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3f197cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:30:01,095-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [181bae2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:30:01,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [181bae2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d2725c1 >2018-05-29 07:30:01,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:30:01,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [181bae2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d2725c1 >2018-05-29 07:30:01,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:01,489-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:01,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:01,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:01,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:01,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [181bae2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:01,501-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [181bae2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:30:01,504-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [181bae2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 181bae2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:30:01,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56748444 >2018-05-29 07:30:01,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:30:01,611-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:30:01,611-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:30:01,611-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56748444 >2018-05-29 07:30:01,611-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:30:01,625-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [a2a837] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:30:01,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [a2a837] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 775dcdd2 >2018-05-29 07:30:01,628-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [a2a837] FINISH, SetVdsStatusVDSCommand, log id: 775dcdd2 >2018-05-29 07:30:01,637-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a2a837, Job ID: 16cafe5a-5d45-48b6-94bd-527f5dbfebf4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:30:01,641-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f197cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:30:01,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [a2a837] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f197cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:30:01,655-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [7aebe8e2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:30:01,656-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [7aebe8e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:30:02,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67c51276] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4289cce9 >2018-05-29 07:30:02,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67c51276] FINISH, GetHardwareInfoVDSCommand, log id: 4289cce9 >2018-05-29 07:30:02,901-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [77fb800c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:30:02,905-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [434907d2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:30:02,911-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6ae9345d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:30:02,918-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6ae9345d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1df6902a >2018-05-29 07:30:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:30:03,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6ae9345d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1df6902a >2018-05-29 07:30:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:03,290-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:03,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:03,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:03,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:30:03,299-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6ae9345d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:30:03,348-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6ae9345d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:30:03,352-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ae9345d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ae9345d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:30:03,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f2daaf0 >2018-05-29 07:30:03,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:30:03,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:30:03,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:30:03,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: f2daaf0 >2018-05-29 07:30:03,460-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:30:03,474-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3b48282c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:30:03,476-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3b48282c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e408f6b >2018-05-29 07:30:03,478-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3b48282c] FINISH, SetVdsStatusVDSCommand, log id: 5e408f6b >2018-05-29 07:30:03,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b48282c, Job ID: e73e14a5-5f50-4f20-ab84-964145c9a570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:30:03,516-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 434907d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:30:03,518-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b48282c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 434907d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:30:03,529-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6efc8832] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:30:03,530-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6efc8832] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [688dca57] Autorecovering 2 hosts >2018-05-29 07:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [688dca57] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [482150a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a749bd >2018-05-29 07:35:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [482150a6] FINISH, SetVdsStatusVDSCommand, log id: 4a749bd >2018-05-29 07:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [482150a6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:35:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [482150a6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12ce44e9 >2018-05-29 07:35:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] FINISH, SetVdsStatusVDSCommand, log id: 12ce44e9 >2018-05-29 07:35:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6291685d >2018-05-29 07:35:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [366ce5c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6291685d >2018-05-29 07:35:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:35:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [366ce5c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:35:01,264-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [aa1d52] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c787353 >2018-05-29 07:35:01,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [aa1d52] FINISH, GetHardwareInfoVDSCommand, log id: c787353 >2018-05-29 07:35:01,279-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3aac99c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:35:01,288-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [26827007] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:35:01,293-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [233c2c27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:35:01,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [233c2c27] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b0d8601 >2018-05-29 07:35:01,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:35:01,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [233c2c27] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b0d8601 >2018-05-29 07:35:01,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:01,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:01,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:01,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:01,692-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:01,692-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [233c2c27] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:01,695-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [233c2c27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:35:01,698-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [233c2c27] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 233c2c27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:35:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 569e0d52 >2018-05-29 07:35:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:35:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:35:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:35:01,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 569e0d52 >2018-05-29 07:35:01,799-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:35:01,813-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [39fa82f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:35:01,814-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [39fa82f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4571374 >2018-05-29 07:35:01,816-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [39fa82f6] FINISH, SetVdsStatusVDSCommand, log id: 4571374 >2018-05-29 07:35:01,849-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39fa82f6, Job ID: 878b3fe2-2fc2-4cad-bb27-2ba88080232a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:35:01,853-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 26827007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:35:01,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39fa82f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 26827007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:35:01,867-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [8dde309] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:35:01,868-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [8dde309] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:35:02,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1a875132] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11d25337 >2018-05-29 07:35:02,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1a875132] FINISH, GetHardwareInfoVDSCommand, log id: 11d25337 >2018-05-29 07:35:02,299-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [43a2967f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:35:02,304-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1e0fb6f9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:35:02,309-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4f7892f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:35:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4f7892f8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5765ae90 >2018-05-29 07:35:02,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:35:02,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4f7892f8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5765ae90 >2018-05-29 07:35:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:02,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:02,700-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:02,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:02,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:35:02,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4f7892f8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:35:02,707-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4f7892f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:35:02,710-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f7892f8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f7892f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:35:02,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 524cdd45 >2018-05-29 07:35:02,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:35:02,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:35:02,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:35:02,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 524cdd45 >2018-05-29 07:35:02,813-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:35:02,825-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [a48f9e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:35:02,826-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a48f9e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 107ab3c >2018-05-29 07:35:02,828-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a48f9e3] FINISH, SetVdsStatusVDSCommand, log id: 107ab3c >2018-05-29 07:35:02,833-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a48f9e3, Job ID: b0abe400-a02c-49f9-bd11-f9e61a6887a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:35:02,838-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e0fb6f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:35:02,841-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a48f9e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e0fb6f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:35:02,851-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [47a53365] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:35:02,853-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [47a53365] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [436fc54e] Autorecovering 2 hosts >2018-05-29 07:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [436fc54e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2db2dcbd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d4e9ed5 >2018-05-29 07:40:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2db2dcbd] FINISH, SetVdsStatusVDSCommand, log id: 5d4e9ed5 >2018-05-29 07:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2db2dcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:40:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2db2dcbd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:40:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ae466e1 >2018-05-29 07:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] FINISH, SetVdsStatusVDSCommand, log id: ae466e1 >2018-05-29 07:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1cf460fb >2018-05-29 07:40:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5b2c45a5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1cf460fb >2018-05-29 07:40:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:40:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5b2c45a5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:40:00,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b2b3187] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e2b4761 >2018-05-29 07:40:00,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b2b3187] FINISH, GetHardwareInfoVDSCommand, log id: 2e2b4761 >2018-05-29 07:40:00,522-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [67d6be8d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:40:00,530-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [27a5811f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:40:00,535-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7fad7981] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:40:00,546-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7fad7981] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17ae0170 >2018-05-29 07:40:00,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:40:00,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7fad7981] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17ae0170 >2018-05-29 07:40:00,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:00,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:00,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:00,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7fad7981] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:00,944-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7fad7981] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:40:00,946-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7fad7981] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fad7981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:40:00,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d6875f0 >2018-05-29 07:40:00,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:40:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:40:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:40:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:40:01,051-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:40:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: d6875f0 >2018-05-29 07:40:01,051-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:40:01,064-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [60e8ecf3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:40:01,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60e8ecf3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42857756 >2018-05-29 07:40:01,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60e8ecf3] FINISH, SetVdsStatusVDSCommand, log id: 42857756 >2018-05-29 07:40:01,100-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60e8ecf3, Job ID: b30bb010-1d8f-4e3f-b90a-722f7f8e7f60, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:40:01,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27a5811f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:40:01,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60e8ecf3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27a5811f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:40:01,118-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4318c10] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:40:01,119-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4318c10] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:40:02,622-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3499ddff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6df770e2 >2018-05-29 07:40:02,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3499ddff] FINISH, GetHardwareInfoVDSCommand, log id: 6df770e2 >2018-05-29 07:40:02,636-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [35f85fa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:40:02,641-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [2de3bded] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:40:02,646-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3242bae9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:40:02,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3242bae9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1af671fa >2018-05-29 07:40:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:40:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3242bae9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1af671fa >2018-05-29 07:40:03,027-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:03,027-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:03,031-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:03,032-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:03,036-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:40:03,037-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3242bae9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:40:03,040-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3242bae9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:40:03,043-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3242bae9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3242bae9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:40:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6e92680a >2018-05-29 07:40:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:40:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:40:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:40:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:40:03,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:40:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6e92680a >2018-05-29 07:40:03,149-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:40:03,163-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5ad07023] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:40:03,164-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5ad07023] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 316b0384 >2018-05-29 07:40:03,166-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5ad07023] FINISH, SetVdsStatusVDSCommand, log id: 316b0384 >2018-05-29 07:40:03,197-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ad07023, Job ID: 2e89fc54-0ca8-40a5-881f-fbee7afe062b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:40:03,202-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2de3bded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:40:03,204-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ad07023] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2de3bded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:40:03,215-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [3be5efe3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:40:03,216-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [3be5efe3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [f61b467] Autorecovering 2 hosts >2018-05-29 07:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [f61b467] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2e633e15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d7beaff >2018-05-29 07:45:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2e633e15] FINISH, SetVdsStatusVDSCommand, log id: 1d7beaff >2018-05-29 07:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2e633e15] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2e633e15] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:45:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 208929f2 >2018-05-29 07:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] FINISH, SetVdsStatusVDSCommand, log id: 208929f2 >2018-05-29 07:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 539e5e45 >2018-05-29 07:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [3b2c60cb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 539e5e45 >2018-05-29 07:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [3b2c60cb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:45:00,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [338949cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e350716 >2018-05-29 07:45:00,880-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [338949cb] FINISH, GetHardwareInfoVDSCommand, log id: 1e350716 >2018-05-29 07:45:00,893-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [32495681] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:45:00,901-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1a2d3308] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:45:00,906-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [70f3b6bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:45:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [70f3b6bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e6dfdd3 >2018-05-29 07:45:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:45:01,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [70f3b6bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e6dfdd3 >2018-05-29 07:45:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:01,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:01,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:01,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:01,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:01,348-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [70f3b6bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:01,351-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [70f3b6bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:45:01,353-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70f3b6bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70f3b6bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:45:01,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 173327e5 >2018-05-29 07:45:01,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:45:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:45:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:45:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 173327e5 >2018-05-29 07:45:01,457-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:45:01,470-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1ff6be63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:45:01,471-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ff6be63] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3245f397 >2018-05-29 07:45:01,474-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ff6be63] FINISH, SetVdsStatusVDSCommand, log id: 3245f397 >2018-05-29 07:45:01,482-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ff6be63, Job ID: 3314a4c2-023e-43d0-9404-7bbb3006caad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:45:01,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a2d3308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:45:01,489-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ff6be63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a2d3308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:45:01,499-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [78dd2f2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:45:01,500-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [78dd2f2d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:45:01,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4d0ccadf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 420a658 >2018-05-29 07:45:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4d0ccadf] FINISH, GetHardwareInfoVDSCommand, log id: 420a658 >2018-05-29 07:45:01,716-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [72b5c834] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:45:01,721-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3b5d337a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:45:01,726-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1514d148] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:45:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1514d148] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e1af551 >2018-05-29 07:45:02,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:45:02,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1514d148] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e1af551 >2018-05-29 07:45:02,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:02,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:02,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:02,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:02,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:45:02,118-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1514d148] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:45:02,121-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1514d148] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:45:02,124-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1514d148] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1514d148, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:45:02,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e9c8834 >2018-05-29 07:45:02,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:45:02,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:45:02,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:45:02,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: e9c8834 >2018-05-29 07:45:02,226-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:45:02,238-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [586c3682] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:45:02,240-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586c3682] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34fefa3c >2018-05-29 07:45:02,242-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586c3682] FINISH, SetVdsStatusVDSCommand, log id: 34fefa3c >2018-05-29 07:45:02,247-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 586c3682, Job ID: 33e05068-2433-4679-adff-4ae39202cb3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:45:02,252-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b5d337a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:45:02,254-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586c3682] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b5d337a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:45:02,265-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [41424396] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:45:02,266-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [41424396] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [536c7eb3] Autorecovering 2 hosts >2018-05-29 07:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [536c7eb3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:50:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [741d8e4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53ed4116 >2018-05-29 07:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [741d8e4a] FINISH, SetVdsStatusVDSCommand, log id: 53ed4116 >2018-05-29 07:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [741d8e4a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:50:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [741d8e4a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:50:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f292117 >2018-05-29 07:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] FINISH, SetVdsStatusVDSCommand, log id: 5f292117 >2018-05-29 07:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2da1be0c >2018-05-29 07:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [7e1f9d08] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2da1be0c >2018-05-29 07:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7e1f9d08] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:50:02,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [129c2af8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 30c684e7 >2018-05-29 07:50:02,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [129c2af8] FINISH, GetHardwareInfoVDSCommand, log id: 30c684e7 >2018-05-29 07:50:02,146-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6a2dc695] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:50:02,154-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [617adc09] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:50:02,159-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eaa84d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:50:02,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eaa84d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 8cb259e >2018-05-29 07:50:02,539-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:50:02,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5eaa84d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 8cb259e >2018-05-29 07:50:02,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:02,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:02,552-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5eaa84d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:02,559-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5eaa84d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:50:02,562-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5eaa84d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5eaa84d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:50:02,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 60662bf5 >2018-05-29 07:50:02,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:50:02,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:50:02,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:50:02,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 60662bf5 >2018-05-29 07:50:02,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:50:02,678-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [39d9df17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:50:02,679-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39d9df17] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76de70 >2018-05-29 07:50:02,681-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39d9df17] FINISH, SetVdsStatusVDSCommand, log id: 76de70 >2018-05-29 07:50:02,712-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39d9df17, Job ID: 85f9f564-ab70-454d-a297-c7c1fba4ff84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:50:02,717-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 617adc09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:50:02,719-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39d9df17] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 617adc09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:50:02,729-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5ec568a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:50:02,730-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5ec568a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:50:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [67fda786] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3a67eec2 >2018-05-29 07:50:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [67fda786] FINISH, GetHardwareInfoVDSCommand, log id: 3a67eec2 >2018-05-29 07:50:03,184-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [442a2317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:50:03,192-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [658f7146] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:50:03,197-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [515274e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:50:03,209-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [515274e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c3a72ae >2018-05-29 07:50:03,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:50:03,598-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [515274e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c3a72ae >2018-05-29 07:50:03,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:03,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:03,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:03,608-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:03,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:50:03,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [515274e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:50:03,615-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [515274e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:50:03,617-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [515274e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 515274e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:50:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e6f5793 >2018-05-29 07:50:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:50:03,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:50:03,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:50:03,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e6f5793 >2018-05-29 07:50:03,723-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:50:03,735-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [65b7046d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:50:03,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [65b7046d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4875b673 >2018-05-29 07:50:03,738-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [65b7046d] FINISH, SetVdsStatusVDSCommand, log id: 4875b673 >2018-05-29 07:50:03,769-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65b7046d, Job ID: a31adbc8-a0b4-4531-950e-51ae8b7efb87, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:50:03,774-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 658f7146, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:50:03,776-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65b7046d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 658f7146, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:50:03,787-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [8649add] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:50:03,788-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [8649add] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:54:38,333-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [469b59d4] Backup check started. >2018-05-29 07:54:38,349-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [469b59d4] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption. >2018-05-29 07:54:38,349-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [469b59d4] Backup check completed. >2018-05-29 07:54:42,906-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [5dc7eaba] Attempting to update VMs/Templates Ovf. >2018-05-29 07:54:42,909-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 07:54:42,909-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 07:54:42,910-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:54:42,960-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 07:54:42,964-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 07:54:42,964-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 07:54:42,964-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 07:54:42,964-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 07:54:42,965-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 07:54:42,966-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [138a01] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 07:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [716dbdf8] Autorecovering 2 hosts >2018-05-29 07:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [716dbdf8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d2b9edd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c270978 >2018-05-29 07:55:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d2b9edd] FINISH, SetVdsStatusVDSCommand, log id: 2c270978 >2018-05-29 07:55:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d2b9edd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 07:55:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d2b9edd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:55:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 07:55:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [63b94db0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9b2f132 >2018-05-29 07:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [63b94db0] FINISH, SetVdsStatusVDSCommand, log id: 9b2f132 >2018-05-29 07:55:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [63b94db0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6caeaffe >2018-05-29 07:55:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [63b94db0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6caeaffe >2018-05-29 07:55:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 07:55:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [63b94db0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 07:55:00,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [74887fad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 15b4ec6e >2018-05-29 07:55:00,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [74887fad] FINISH, GetHardwareInfoVDSCommand, log id: 15b4ec6e >2018-05-29 07:55:00,405-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e7c96bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:55:00,410-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [730ea822] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:55:00,415-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [60e5861d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:55:00,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [60e5861d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c6a8ae0 >2018-05-29 07:55:00,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:55:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [60e5861d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c6a8ae0 >2018-05-29 07:55:00,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:00,817-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:00,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:00,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:00,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:00,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [60e5861d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:00,830-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [60e5861d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:55:00,832-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [60e5861d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 60e5861d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:55:00,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 676552ce >2018-05-29 07:55:00,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:55:00,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:55:00,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:55:00,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 676552ce >2018-05-29 07:55:00,938-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:55:00,952-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f98df48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:55:00,954-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f98df48] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a829703 >2018-05-29 07:55:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f98df48] FINISH, SetVdsStatusVDSCommand, log id: 5a829703 >2018-05-29 07:55:00,964-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f98df48, Job ID: 14d40912-6dbd-4b96-b620-af63f7765b0e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:55:00,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 730ea822, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:55:00,971-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f98df48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 730ea822, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:55:00,982-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [312353c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 07:55:00,983-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [312353c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 07:55:01,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5fb70c89] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c51ee27 >2018-05-29 07:55:01,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5fb70c89] FINISH, GetHardwareInfoVDSCommand, log id: 7c51ee27 >2018-05-29 07:55:01,422-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [79c4e81c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:55:01,427-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [48536b05] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:55:01,432-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e6c8610] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 07:55:01,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e6c8610] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 304c5f88 >2018-05-29 07:55:01,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 07:55:01,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e6c8610] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 304c5f88 >2018-05-29 07:55:01,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:01,819-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:01,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:01,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:01,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 07:55:01,829-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e6c8610] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 07:55:01,831-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e6c8610] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 07:55:01,834-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e6c8610] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e6c8610, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 07:55:01,838-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2e8b74 >2018-05-29 07:55:01,838-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:55:01,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 07:55:01,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 07:55:01,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2e8b74 >2018-05-29 07:55:01,938-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 07:55:01,952-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [110c450] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:55:01,953-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [110c450] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3442b3e8 >2018-05-29 07:55:01,956-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [110c450] FINISH, SetVdsStatusVDSCommand, log id: 3442b3e8 >2018-05-29 07:55:01,961-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 110c450, Job ID: 9b71602c-515f-4ca2-86d1-90213ec6f941, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 07:55:01,966-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 48536b05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 07:55:01,968-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [110c450] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 48536b05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 07:55:01,980-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [18eb1ce3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 07:55:01,981-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [18eb1ce3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [ab9a723] Autorecovering 2 hosts >2018-05-29 08:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [ab9a723] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:00:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [510ca8b4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 477d7e37 >2018-05-29 08:00:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [510ca8b4] FINISH, SetVdsStatusVDSCommand, log id: 477d7e37 >2018-05-29 08:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [510ca8b4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:00:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [510ca8b4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:00:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [435c937c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31edd54 >2018-05-29 08:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [435c937c] FINISH, SetVdsStatusVDSCommand, log id: 31edd54 >2018-05-29 08:00:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [435c937c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58848b14 >2018-05-29 08:00:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [435c937c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58848b14 >2018-05-29 08:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [435c937c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:00:01,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [c4576e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24939a5a >2018-05-29 08:00:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [c4576e6] FINISH, GetHardwareInfoVDSCommand, log id: 24939a5a >2018-05-29 08:00:01,643-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [75102a3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:00:01,651-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5e980b66] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:00:01,656-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [1ccb8db7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:00:01,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [1ccb8db7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4deb458f >2018-05-29 08:00:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:00:02,038-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [1ccb8db7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4deb458f >2018-05-29 08:00:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:02,042-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:02,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:02,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:02,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:02,051-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [1ccb8db7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:02,054-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [1ccb8db7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:00:02,057-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ccb8db7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ccb8db7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:00:02,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 58ff4e33 >2018-05-29 08:00:02,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:00:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:00:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:00:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 58ff4e33 >2018-05-29 08:00:02,159-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:00:02,172-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [31f5a989] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:00:02,173-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31f5a989] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a0e17a >2018-05-29 08:00:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31f5a989] FINISH, SetVdsStatusVDSCommand, log id: 6a0e17a >2018-05-29 08:00:02,207-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31f5a989, Job ID: e333a530-e8f6-4b1c-8acc-f32f1262251d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:00:02,211-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e980b66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:00:02,213-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [31f5a989] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e980b66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:00:02,224-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2840a4c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:00:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2840a4c0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:00:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [68951fb1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 47d7e11b >2018-05-29 08:00:02,612-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [68951fb1] FINISH, GetHardwareInfoVDSCommand, log id: 47d7e11b >2018-05-29 08:00:02,624-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5ffc0f50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:00:02,629-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [15650649] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:00:02,634-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [23b47465] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:00:02,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [23b47465] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 44c15250 >2018-05-29 08:00:03,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:00:03,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [23b47465] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 44c15250 >2018-05-29 08:00:03,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:03,017-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:03,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:03,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:03,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:00:03,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [23b47465] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:00:03,029-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [23b47465] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:00:03,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23b47465] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23b47465, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:00:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 675112f1 >2018-05-29 08:00:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:00:03,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:00:03,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:00:03,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 675112f1 >2018-05-29 08:00:03,132-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:00:03,145-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7b25b152] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:00:03,146-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7b25b152] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 115eaad0 >2018-05-29 08:00:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7b25b152] FINISH, SetVdsStatusVDSCommand, log id: 115eaad0 >2018-05-29 08:00:03,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7b25b152, Job ID: 2c7a30f0-a43a-4324-91d6-620ef69eee9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:00:03,158-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15650649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:00:03,160-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7b25b152] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15650649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:00:03,170-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [459c10cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:00:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [459c10cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6cd5b6d2] Autorecovering 2 hosts >2018-05-29 08:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6cd5b6d2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:05:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:05:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:05:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [248de92c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4eb14dd9 >2018-05-29 08:05:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [248de92c] FINISH, SetVdsStatusVDSCommand, log id: 4eb14dd9 >2018-05-29 08:05:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [248de92c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:05:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [248de92c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:05:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 615bcde2 >2018-05-29 08:05:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] FINISH, SetVdsStatusVDSCommand, log id: 615bcde2 >2018-05-29 08:05:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 21dae6f0 >2018-05-29 08:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6a95de9e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 21dae6f0 >2018-05-29 08:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6a95de9e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:05:00,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2fc9ef36] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e1f464d >2018-05-29 08:05:00,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2fc9ef36] FINISH, GetHardwareInfoVDSCommand, log id: 7e1f464d >2018-05-29 08:05:00,817-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5f76b131] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:05:00,826-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [20c3fb7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:05:00,831-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [583083bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:05:00,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [583083bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a322c80 >2018-05-29 08:05:01,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:05:01,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [583083bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a322c80 >2018-05-29 08:05:01,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:01,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:01,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:01,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:01,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:01,243-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [583083bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:01,246-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [583083bf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:05:01,249-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [583083bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 583083bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:05:01,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3013cfde >2018-05-29 08:05:01,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:05:01,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:05:01,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:05:01,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:05:01,354-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:05:01,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3013cfde >2018-05-29 08:05:01,355-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:05:01,368-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [587151f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:05:01,369-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [587151f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 408c0eff >2018-05-29 08:05:01,371-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [587151f6] FINISH, SetVdsStatusVDSCommand, log id: 408c0eff >2018-05-29 08:05:01,402-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 587151f6, Job ID: 38ef46f2-2e69-4959-a88a-d38a900c7c72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:05:01,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20c3fb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:05:01,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [587151f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20c3fb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:05:01,419-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f521be2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:05:01,420-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f521be2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:05:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [9ad4c5e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7ea19ea >2018-05-29 08:05:02,911-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [9ad4c5e] FINISH, GetHardwareInfoVDSCommand, log id: 7ea19ea >2018-05-29 08:05:02,924-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5d2e6790] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:05:02,929-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [29918fb8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:05:02,934-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [44d9d918] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:05:02,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [44d9d918] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3dc1270a >2018-05-29 08:05:03,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:05:03,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [44d9d918] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3dc1270a >2018-05-29 08:05:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:03,319-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:03,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:03,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:05:03,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [44d9d918] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:05:03,331-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [44d9d918] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:05:03,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44d9d918] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 44d9d918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:05:03,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76916ebc >2018-05-29 08:05:03,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:05:03,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:05:03,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:05:03,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76916ebc >2018-05-29 08:05:03,436-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:05:03,450-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d308030] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:05:03,451-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d308030] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3725152 >2018-05-29 08:05:03,453-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d308030] FINISH, SetVdsStatusVDSCommand, log id: 3725152 >2018-05-29 08:05:03,484-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d308030, Job ID: da218389-8855-46ae-a8d0-dc5a79d4e68d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:05:03,488-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29918fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:05:03,491-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d308030] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29918fb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:05:03,502-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [757991a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:05:03,503-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [757991a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57bbf925] Autorecovering 2 hosts >2018-05-29 08:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [57bbf925] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4cb179b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ee0fc94 >2018-05-29 08:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4cb179b7] FINISH, SetVdsStatusVDSCommand, log id: 1ee0fc94 >2018-05-29 08:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4cb179b7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4cb179b7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50f67730] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b9e1ded >2018-05-29 08:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50f67730] FINISH, SetVdsStatusVDSCommand, log id: b9e1ded >2018-05-29 08:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [50f67730] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51304bef >2018-05-29 08:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [50f67730] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51304bef >2018-05-29 08:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50f67730] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:10:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3304b684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d14e98d >2018-05-29 08:10:01,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3304b684] FINISH, GetHardwareInfoVDSCommand, log id: d14e98d >2018-05-29 08:10:01,230-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [ae2f096] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:10:01,239-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [552590d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:10:01,244-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5ce86f73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:10:01,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5ce86f73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a101daf >2018-05-29 08:10:02,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2b3dfe35] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 512aab9d >2018-05-29 08:10:02,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2b3dfe35] FINISH, GetHardwareInfoVDSCommand, log id: 512aab9d >2018-05-29 08:10:02,106-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [2d97499a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:10:02,110-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5098c04d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:10:02,115-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4d0734cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:10:02,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4d0734cd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30a4f4a1 >2018-05-29 08:10:02,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:10:02,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4d0734cd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30a4f4a1 >2018-05-29 08:10:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4d0734cd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,521-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4d0734cd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:10:02,523-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d0734cd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d0734cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:10:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57bda063 >2018-05-29 08:10:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:10:02,618-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:10:02,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5ce86f73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a101daf >2018-05-29 08:10:02,622-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,627-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:10:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:10:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:10:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57bda063 >2018-05-29 08:10:02,628-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:10:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:10:02,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5ce86f73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:10:02,636-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5ce86f73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:10:02,639-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5ce86f73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ce86f73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:10:02,641-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1eee2ba3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:10:02,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 388daeab >2018-05-29 08:10:02,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:10:02,642-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1eee2ba3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e54788c >2018-05-29 08:10:02,645-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1eee2ba3] FINISH, SetVdsStatusVDSCommand, log id: 5e54788c >2018-05-29 08:10:02,677-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1eee2ba3, Job ID: 74adbddf-dacb-48f5-aac0-702e102527ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:10:02,682-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5098c04d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:10:02,684-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1eee2ba3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5098c04d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:10:02,695-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4684d7d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:10:02,696-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4684d7d8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:10:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:10:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:10:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 388daeab >2018-05-29 08:10:02,742-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:10:02,754-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [72c264e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:10:02,755-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [72c264e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11d4d6 >2018-05-29 08:10:02,757-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [72c264e2] FINISH, SetVdsStatusVDSCommand, log id: 11d4d6 >2018-05-29 08:10:02,762-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72c264e2, Job ID: dc80abd9-a0dd-4523-8dd6-ae3789566ba6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:10:02,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 552590d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:10:02,769-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [72c264e2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 552590d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:10:02,779-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1e905ec8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:10:02,780-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1e905ec8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [25cf493e] Autorecovering 2 hosts >2018-05-29 08:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [25cf493e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7bc6240c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6cfc70ac >2018-05-29 08:15:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7bc6240c] FINISH, SetVdsStatusVDSCommand, log id: 6cfc70ac >2018-05-29 08:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7bc6240c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:15:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7bc6240c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:15:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19c4a6cf >2018-05-29 08:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] FINISH, SetVdsStatusVDSCommand, log id: 19c4a6cf >2018-05-29 08:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 765280a1 >2018-05-29 08:15:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22cff3b6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 765280a1 >2018-05-29 08:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:15:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22cff3b6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:15:00,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1045ea68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5d02b4e9 >2018-05-29 08:15:00,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1045ea68] FINISH, GetHardwareInfoVDSCommand, log id: 5d02b4e9 >2018-05-29 08:15:00,402-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7509f81a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:15:00,411-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [99edcf8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:15:00,417-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77ed9772] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:15:00,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77ed9772] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f4baaa9 >2018-05-29 08:15:00,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [66dc9526] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 78b81222 >2018-05-29 08:15:00,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [66dc9526] FINISH, GetHardwareInfoVDSCommand, log id: 78b81222 >2018-05-29 08:15:00,551-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [13e68c51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:15:00,555-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [32979254] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:15:00,560-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6871fc02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:15:00,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6871fc02] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e7bc0eb >2018-05-29 08:15:00,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:15:00,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77ed9772] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f4baaa9 >2018-05-29 08:15:00,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77ed9772] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,877-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77ed9772] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:15:00,880-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77ed9772] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77ed9772, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:15:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2fd7b5cf >2018-05-29 08:15:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:15:00,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:15:00,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6871fc02] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e7bc0eb >2018-05-29 08:15:00,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:15:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6871fc02] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:15:00,953-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6871fc02] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:15:00,956-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6871fc02] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6871fc02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:15:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:15:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:15:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2fd7b5cf >2018-05-29 08:15:00,987-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:15:00,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1fef81f7 >2018-05-29 08:15:00,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:15:01,000-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5f5491b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:15:01,002-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f5491b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f8e1e9 >2018-05-29 08:15:01,004-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5f5491b7] FINISH, SetVdsStatusVDSCommand, log id: f8e1e9 >2018-05-29 08:15:01,035-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f5491b7, Job ID: dc39eba3-5707-4e34-8500-eb8bc6897d3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:15:01,040-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 99edcf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:15:01,042-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f5491b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 99edcf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:15:01,053-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5f2cb41b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:15:01,054-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5f2cb41b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:15:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:15:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:15:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1fef81f7 >2018-05-29 08:15:01,087-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:15:01,101-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [226d75] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:15:01,102-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226d75] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e49d1f2 >2018-05-29 08:15:01,104-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226d75] FINISH, SetVdsStatusVDSCommand, log id: 7e49d1f2 >2018-05-29 08:15:01,110-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 226d75, Job ID: 1e7ab2f3-8959-4882-8f5b-4a8786a86d1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:15:01,114-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 32979254, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:15:01,117-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226d75] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 32979254, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:15:01,127-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4cc00193] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:15:01,128-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4cc00193] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4d7b53ad] Autorecovering 2 hosts >2018-05-29 08:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4d7b53ad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:20:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:20:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:20:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [72dec653] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5fdf15b5 >2018-05-29 08:20:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [72dec653] FINISH, SetVdsStatusVDSCommand, log id: 5fdf15b5 >2018-05-29 08:20:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [72dec653] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:20:00,058-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [72dec653] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:20:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6db25422] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5295a716 >2018-05-29 08:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6db25422] FINISH, SetVdsStatusVDSCommand, log id: 5295a716 >2018-05-29 08:20:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6db25422] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6874a49b >2018-05-29 08:20:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [6db25422] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6874a49b >2018-05-29 08:20:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6db25422] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:20:01,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4027c379] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 759fa88c >2018-05-29 08:20:01,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4027c379] FINISH, GetHardwareInfoVDSCommand, log id: 759fa88c >2018-05-29 08:20:01,685-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3075636d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:20:01,693-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7f9891dc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:20:01,698-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [49f07ee9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:20:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [49f07ee9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 715b5d89 >2018-05-29 08:20:01,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [31354382] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 29200aa8 >2018-05-29 08:20:01,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [31354382] FINISH, GetHardwareInfoVDSCommand, log id: 29200aa8 >2018-05-29 08:20:01,769-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4cf1fbf8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:20:01,774-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [103c8543] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:20:01,778-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75b6cd61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:20:01,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75b6cd61] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c2e30 >2018-05-29 08:20:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:20:02,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [49f07ee9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 715b5d89 >2018-05-29 08:20:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,085-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,093-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,094-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [49f07ee9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,096-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [49f07ee9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:20:02,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49f07ee9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49f07ee9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:20:02,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5abbc84b >2018-05-29 08:20:02,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:20:02,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:20:02,152-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75b6cd61] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c2e30 >2018-05-29 08:20:02,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:20:02,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75b6cd61] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:20:02,168-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75b6cd61] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:20:02,170-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75b6cd61] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75b6cd61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:20:02,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:20:02,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:20:02,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5abbc84b >2018-05-29 08:20:02,203-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:20:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31e8df0d >2018-05-29 08:20:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:20:02,216-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [49da411] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:20:02,217-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [49da411] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e8faefb >2018-05-29 08:20:02,219-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [49da411] FINISH, SetVdsStatusVDSCommand, log id: 3e8faefb >2018-05-29 08:20:02,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49da411, Job ID: 98df4ce9-9100-42e9-bd3d-56c193a4c18e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:20:02,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f9891dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:20:02,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [49da411] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f9891dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:20:02,267-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [22cab21f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:20:02,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [22cab21f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:20:02,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:20:02,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:20:02,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31e8df0d >2018-05-29 08:20:02,302-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:20:02,314-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [658a78a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:20:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [658a78a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24b11df3 >2018-05-29 08:20:02,317-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [658a78a4] FINISH, SetVdsStatusVDSCommand, log id: 24b11df3 >2018-05-29 08:20:02,323-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 658a78a4, Job ID: c384ac7e-920c-4401-ac96-0210e15757ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:20:02,327-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 103c8543, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:20:02,329-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [658a78a4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 103c8543, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:20:02,340-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [16e99fc7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:20:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [16e99fc7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [984de79] Autorecovering 2 hosts >2018-05-29 08:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [984de79] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [139ca2ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42b2437e >2018-05-29 08:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [139ca2ec] FINISH, SetVdsStatusVDSCommand, log id: 42b2437e >2018-05-29 08:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [139ca2ec] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:25:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [139ca2ec] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7389598f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18f541f >2018-05-29 08:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7389598f] FINISH, SetVdsStatusVDSCommand, log id: 18f541f >2018-05-29 08:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7389598f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6de603ed >2018-05-29 08:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7389598f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6de603ed >2018-05-29 08:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:25:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7389598f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:25:03,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [ee613cf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a56bf >2018-05-29 08:25:03,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [ee613cf] FINISH, GetHardwareInfoVDSCommand, log id: 6a56bf >2018-05-29 08:25:03,719-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [6f136aa1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:25:03,727-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4a019873] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:25:03,732-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1545f8e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:25:03,743-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1545f8e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 159c5d45 >2018-05-29 08:25:03,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [41f7ac0e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 419a54b3 >2018-05-29 08:25:03,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [41f7ac0e] FINISH, GetHardwareInfoVDSCommand, log id: 419a54b3 >2018-05-29 08:25:03,800-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [36f33edf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:25:03,804-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4ace637] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:25:03,809-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9c28303] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:25:03,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9c28303] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78ca2221 >2018-05-29 08:25:04,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:25:04,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [1545f8e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 159c5d45 >2018-05-29 08:25:04,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,128-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [1545f8e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,140-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [1545f8e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:25:04,143-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1545f8e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1545f8e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:25:04,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53c9da08 >2018-05-29 08:25:04,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:25:04,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:25:04,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9c28303] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78ca2221 >2018-05-29 08:25:04,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,197-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:25:04,202-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9c28303] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:25:04,205-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9c28303] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:25:04,207-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9c28303] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9c28303, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:25:04,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:25:04,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:25:04,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53c9da08 >2018-05-29 08:25:04,265-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:25:04,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5412ed7a >2018-05-29 08:25:04,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:25:04,279-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3ecc6359] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:25:04,280-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ecc6359] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ca50e2f >2018-05-29 08:25:04,282-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ecc6359] FINISH, SetVdsStatusVDSCommand, log id: 2ca50e2f >2018-05-29 08:25:04,314-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ecc6359, Job ID: 7d739f02-0dfd-4c6d-b1a8-32d7769afe02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:25:04,318-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a019873, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:25:04,320-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3ecc6359] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a019873, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:25:04,331-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [782dee07] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:25:04,332-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [782dee07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:25:04,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:25:04,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:25:04,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5412ed7a >2018-05-29 08:25:04,370-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:25:04,383-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1f0b6a79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:25:04,385-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1f0b6a79] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ded7bb4 >2018-05-29 08:25:04,387-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1f0b6a79] FINISH, SetVdsStatusVDSCommand, log id: 2ded7bb4 >2018-05-29 08:25:04,392-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f0b6a79, Job ID: 9307b87a-00de-4ed4-9f5a-f46beffb0da7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:25:04,397-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4ace637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:25:04,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1f0b6a79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4ace637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:25:04,409-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7358c820] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:25:04,410-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7358c820] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [617f4d1d] Autorecovering 2 hosts >2018-05-29 08:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [617f4d1d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [258f6b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72c26b77 >2018-05-29 08:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [258f6b9b] FINISH, SetVdsStatusVDSCommand, log id: 72c26b77 >2018-05-29 08:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [258f6b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [258f6b9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f565838 >2018-05-29 08:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] FINISH, SetVdsStatusVDSCommand, log id: 3f565838 >2018-05-29 08:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56c63ca0 >2018-05-29 08:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3ac085b5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56c63ca0 >2018-05-29 08:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3ac085b5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:30:02,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12bc242c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7613b19e >2018-05-29 08:30:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12bc242c] FINISH, GetHardwareInfoVDSCommand, log id: 7613b19e >2018-05-29 08:30:02,019-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5ba8d006] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:30:02,027-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [69a58ca1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:30:02,032-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4802da5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:30:02,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4802da5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: df9ebbc >2018-05-29 08:30:02,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3c2b7921] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 303c8da >2018-05-29 08:30:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3c2b7921] FINISH, GetHardwareInfoVDSCommand, log id: 303c8da >2018-05-29 08:30:02,069-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6f35ce96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:30:02,073-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2e616128] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:30:02,078-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [32d9495c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:30:02,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [32d9495c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e4e754 >2018-05-29 08:30:02,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:30:02,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4802da5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: df9ebbc >2018-05-29 08:30:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:02,431-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:02,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:02,435-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:02,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:02,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4802da5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:02,442-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4802da5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:30:02,445-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4802da5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4802da5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:30:02,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 496bd3d >2018-05-29 08:30:02,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:30:02,550-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:30:02,550-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:30:02,550-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 496bd3d >2018-05-29 08:30:02,550-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:30:02,563-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [26c2ce21] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:30:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [26c2ce21] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35e8d350 >2018-05-29 08:30:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [26c2ce21] FINISH, SetVdsStatusVDSCommand, log id: 35e8d350 >2018-05-29 08:30:02,597-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 26c2ce21, Job ID: d0f2eaeb-1023-4965-a191-920aa961e3bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:30:02,602-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69a58ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:30:02,605-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26c2ce21] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69a58ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:30:02,615-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [12432a4a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:30:02,616-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [12432a4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:30:03,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:30:03,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [32d9495c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e4e754 >2018-05-29 08:30:03,456-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:03,457-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:03,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:03,461-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:03,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:30:03,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [32d9495c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:30:03,469-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [32d9495c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:30:03,471-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32d9495c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 32d9495c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:30:03,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57fb8dd6 >2018-05-29 08:30:03,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:30:03,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:30:03,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:30:03,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57fb8dd6 >2018-05-29 08:30:03,571-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:30:03,583-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [38afc133] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:30:03,584-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38afc133] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1181ea1a >2018-05-29 08:30:03,587-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38afc133] FINISH, SetVdsStatusVDSCommand, log id: 1181ea1a >2018-05-29 08:30:03,616-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38afc133, Job ID: b8655e6b-6879-427c-a0f2-094fa1dac7ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:30:03,621-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e616128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:30:03,623-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38afc133] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e616128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:30:03,634-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [66db3f88] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:30:03,635-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [66db3f88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3995d9a1] Autorecovering 2 hosts >2018-05-29 08:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3995d9a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383d4d67] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a8a81f2 >2018-05-29 08:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383d4d67] FINISH, SetVdsStatusVDSCommand, log id: 5a8a81f2 >2018-05-29 08:35:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [383d4d67] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:35:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [383d4d67] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [d8818d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 572f5307 >2018-05-29 08:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [d8818d1] FINISH, SetVdsStatusVDSCommand, log id: 572f5307 >2018-05-29 08:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [d8818d1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6c4509f8 >2018-05-29 08:35:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [d8818d1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6c4509f8 >2018-05-29 08:35:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:35:00,102-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [d8818d1] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: d8818d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 08:35:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [d8818d1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:35:00,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [58d6eb99] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11e619ef >2018-05-29 08:35:00,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [58d6eb99] FINISH, GetHardwareInfoVDSCommand, log id: 11e619ef >2018-05-29 08:35:00,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [736bfff3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:35:00,238-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [3b658777] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:35:00,243-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75e9226e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:35:00,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75e9226e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 90bb794 >2018-05-29 08:35:00,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:35:00,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75e9226e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 90bb794 >2018-05-29 08:35:00,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:00,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:00,692-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:00,693-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:00,696-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:00,697-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75e9226e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:00,700-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75e9226e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:35:00,702-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75e9226e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75e9226e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:35:00,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c43c021 >2018-05-29 08:35:00,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:35:00,807-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:35:00,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:35:00,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:35:00,807-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:35:00,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: c43c021 >2018-05-29 08:35:00,808-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:35:00,821-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [27fac8b1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:35:00,822-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [27fac8b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7126a167 >2018-05-29 08:35:00,824-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [27fac8b1] FINISH, SetVdsStatusVDSCommand, log id: 7126a167 >2018-05-29 08:35:00,856-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27fac8b1, Job ID: be380688-4df4-42b8-b9b3-3c3ac2d79315, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:35:00,861-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b658777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:35:00,863-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [27fac8b1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b658777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:35:00,874-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5ff917] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:35:00,875-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5ff917] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:35:01,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7eb59dce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 455ee8a1 >2018-05-29 08:35:01,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7eb59dce] FINISH, GetHardwareInfoVDSCommand, log id: 455ee8a1 >2018-05-29 08:35:01,301-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1658bae0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:35:01,305-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [10d3a862] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:35:01,310-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [424fb27e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:35:01,317-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [424fb27e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11a624bf >2018-05-29 08:35:01,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:35:01,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [424fb27e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11a624bf >2018-05-29 08:35:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:01,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:01,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:01,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:35:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [424fb27e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:35:01,707-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [424fb27e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:35:01,710-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [424fb27e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 424fb27e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:35:01,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7533dfe2 >2018-05-29 08:35:01,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:35:01,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:35:01,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:35:01,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7533dfe2 >2018-05-29 08:35:01,812-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:35:01,825-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1ec5209d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:35:01,827-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ec5209d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 386fa481 >2018-05-29 08:35:01,829-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1ec5209d] FINISH, SetVdsStatusVDSCommand, log id: 386fa481 >2018-05-29 08:35:01,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ec5209d, Job ID: dd818572-2f60-4aea-8d6b-39dc165af773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:35:01,839-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10d3a862, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:35:01,841-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ec5209d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10d3a862, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:35:01,851-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1472accc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:35:01,853-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1472accc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [43027f99] Autorecovering 2 hosts >2018-05-29 08:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [43027f99] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a699e85] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556cc032 >2018-05-29 08:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a699e85] FINISH, SetVdsStatusVDSCommand, log id: 556cc032 >2018-05-29 08:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7a699e85] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:40:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a699e85] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:40:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7e970117] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 156aecff >2018-05-29 08:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7e970117] FINISH, SetVdsStatusVDSCommand, log id: 156aecff >2018-05-29 08:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7e970117] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 334a4d62 >2018-05-29 08:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7e970117] FINISH, SetHaMaintenanceModeVDSCommand, log id: 334a4d62 >2018-05-29 08:40:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:40:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7e970117] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:40:01,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [475b6bda] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 66f5bd9e >2018-05-29 08:40:01,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [475b6bda] FINISH, GetHardwareInfoVDSCommand, log id: 66f5bd9e >2018-05-29 08:40:01,612-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55fd7d15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:40:01,620-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [688d2129] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:40:01,625-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7dabb868] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:40:01,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7dabb868] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 164b182 >2018-05-29 08:40:02,022-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:40:02,022-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7dabb868] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 164b182 >2018-05-29 08:40:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:02,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:02,031-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:02,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:02,035-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7dabb868] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:02,038-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7dabb868] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:40:02,041-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7dabb868] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7dabb868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:40:02,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 277f7849 >2018-05-29 08:40:02,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:40:02,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:40:02,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:40:02,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 277f7849 >2018-05-29 08:40:02,146-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:40:02,161-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1776bf79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:40:02,162-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1776bf79] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 363ff07e >2018-05-29 08:40:02,164-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1776bf79] FINISH, SetVdsStatusVDSCommand, log id: 363ff07e >2018-05-29 08:40:02,196-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1776bf79, Job ID: 1fb5acb2-c4a5-4fcb-a2ba-ce415f6793df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:40:02,200-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 688d2129, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:40:02,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1776bf79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 688d2129, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:40:02,214-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [42545303] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:40:02,215-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [42545303] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:40:02,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6ef8caad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b7c49e2 >2018-05-29 08:40:02,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6ef8caad] FINISH, GetHardwareInfoVDSCommand, log id: 6b7c49e2 >2018-05-29 08:40:02,544-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6a145be6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:40:02,548-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3d588d26] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:40:02,553-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [13378390] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:40:02,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [13378390] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6534c065 >2018-05-29 08:40:03,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:40:03,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [13378390] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6534c065 >2018-05-29 08:40:03,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:03,933-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:03,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:03,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:03,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:40:03,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [13378390] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:40:03,945-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [13378390] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:40:03,947-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [13378390] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13378390, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:40:03,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 79d4d4b6 >2018-05-29 08:40:03,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:40:04,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:40:04,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:40:04,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 79d4d4b6 >2018-05-29 08:40:04,048-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:40:04,061-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6ebca3bc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:40:04,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6ebca3bc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7194066d >2018-05-29 08:40:04,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6ebca3bc] FINISH, SetVdsStatusVDSCommand, log id: 7194066d >2018-05-29 08:40:04,070-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6ebca3bc, Job ID: 7fa1fdf6-09fb-4cd5-9ef3-1b11677df328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:40:04,074-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d588d26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:40:04,076-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6ebca3bc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d588d26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:40:04,087-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6c0f2fa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:40:04,088-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6c0f2fa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4f3e3e95] Autorecovering 2 hosts >2018-05-29 08:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4f3e3e95] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70933836] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ae2aa55 >2018-05-29 08:45:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70933836] FINISH, SetVdsStatusVDSCommand, log id: 6ae2aa55 >2018-05-29 08:45:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70933836] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:45:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [70933836] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:45:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:45:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [334bb122] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40baed79 >2018-05-29 08:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [334bb122] FINISH, SetVdsStatusVDSCommand, log id: 40baed79 >2018-05-29 08:45:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [334bb122] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 660f0335 >2018-05-29 08:45:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [334bb122] FINISH, SetHaMaintenanceModeVDSCommand, log id: 660f0335 >2018-05-29 08:45:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:45:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [334bb122] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:45:01,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [e0c6b78] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20e223f8 >2018-05-29 08:45:01,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [e0c6b78] FINISH, GetHardwareInfoVDSCommand, log id: 20e223f8 >2018-05-29 08:45:01,823-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1eecd5e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:45:01,832-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [12dfbeec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:45:01,837-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [ab57446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:45:01,849-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [ab57446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 212bb142 >2018-05-29 08:45:02,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:45:02,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [ab57446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 212bb142 >2018-05-29 08:45:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:02,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:02,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:02,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:02,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:02,238-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [ab57446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:02,241-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [ab57446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:45:02,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [ab57446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ab57446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:45:02,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4fec5132 >2018-05-29 08:45:02,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:45:02,350-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:45:02,350-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:45:02,350-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4fec5132 >2018-05-29 08:45:02,350-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:45:02,364-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1db22768] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:45:02,365-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1db22768] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cbcd21e >2018-05-29 08:45:02,368-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1db22768] FINISH, SetVdsStatusVDSCommand, log id: 3cbcd21e >2018-05-29 08:45:02,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1db22768, Job ID: 77935ae6-043b-4a0b-bd16-e2991106d975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:45:02,404-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 12dfbeec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:45:02,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1db22768] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 12dfbeec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:45:02,417-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1affdb96] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:45:02,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1affdb96] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:45:03,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7659437e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24170f54 >2018-05-29 08:45:03,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7659437e] FINISH, GetHardwareInfoVDSCommand, log id: 24170f54 >2018-05-29 08:45:03,024-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1a041b4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:45:03,029-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5a228065] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:45:03,034-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3362a530] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:45:03,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3362a530] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 554a4b5b >2018-05-29 08:45:03,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:45:03,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3362a530] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 554a4b5b >2018-05-29 08:45:03,420-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:03,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:45:03,430-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3362a530] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:45:03,433-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3362a530] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:45:03,435-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3362a530] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3362a530, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:45:03,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 728914df >2018-05-29 08:45:03,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:45:03,535-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:45:03,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:45:03,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:45:03,535-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:45:03,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 728914df >2018-05-29 08:45:03,536-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:45:03,549-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [e544d16] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:45:03,550-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e544d16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50572791 >2018-05-29 08:45:03,552-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e544d16] FINISH, SetVdsStatusVDSCommand, log id: 50572791 >2018-05-29 08:45:03,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e544d16, Job ID: b6071715-4686-496e-be7c-308ad4a834d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:45:03,562-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a228065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:45:03,565-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [e544d16] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a228065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:45:03,575-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [267ba407] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:45:03,576-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [267ba407] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:50:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2b0a949e] Autorecovering 2 hosts >2018-05-29 08:50:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2b0a949e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:50:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:50:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:50:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1aafcefe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3490ef02 >2018-05-29 08:50:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1aafcefe] FINISH, SetVdsStatusVDSCommand, log id: 3490ef02 >2018-05-29 08:50:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1aafcefe] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:50:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1aafcefe] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:50:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:50:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:50:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3af55209 >2018-05-29 08:50:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] FINISH, SetVdsStatusVDSCommand, log id: 3af55209 >2018-05-29 08:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3638e1b1 >2018-05-29 08:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [3b8c8dde] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3638e1b1 >2018-05-29 08:50:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:50:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3b8c8dde] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:50:01,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [b155955] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d73f372 >2018-05-29 08:50:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [b155955] FINISH, GetHardwareInfoVDSCommand, log id: 4d73f372 >2018-05-29 08:50:01,215-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [78cbd282] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:50:01,220-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [163cabd8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:50:01,225-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3593c75b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:50:01,238-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3593c75b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c1365a3 >2018-05-29 08:50:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:50:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3593c75b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c1365a3 >2018-05-29 08:50:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:01,627-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:01,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:01,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:01,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:01,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3593c75b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:01,639-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3593c75b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:50:01,642-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3593c75b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3593c75b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:50:01,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d3c131d >2018-05-29 08:50:01,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:50:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:50:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:50:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d3c131d >2018-05-29 08:50:01,748-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:50:01,761-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [23e7c5f7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:50:01,763-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [23e7c5f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aa9d24e >2018-05-29 08:50:01,765-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [23e7c5f7] FINISH, SetVdsStatusVDSCommand, log id: 5aa9d24e >2018-05-29 08:50:01,796-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23e7c5f7, Job ID: fac3304a-f2e0-4d5e-854b-d9737e0df58e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:50:01,801-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 163cabd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:50:01,803-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [23e7c5f7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 163cabd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:50:01,813-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [6064e00f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:50:01,815-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6064e00f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:50:03,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2cd282d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5474c9ca >2018-05-29 08:50:03,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2cd282d] FINISH, GetHardwareInfoVDSCommand, log id: 5474c9ca >2018-05-29 08:50:03,180-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1801f6f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:50:03,184-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [ed20748] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:50:03,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [93413b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:50:03,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [93413b9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78bde303 >2018-05-29 08:50:04,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:50:04,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [93413b9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78bde303 >2018-05-29 08:50:04,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:04,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:04,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:04,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:04,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:50:04,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [93413b9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:50:04,580-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [93413b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:50:04,583-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [93413b9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 93413b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:50:04,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 785b793a >2018-05-29 08:50:04,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:50:04,685-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:50:04,685-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:50:04,685-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 785b793a >2018-05-29 08:50:04,685-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:50:04,698-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [358e5112] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:50:04,700-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [358e5112] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 349a8bd4 >2018-05-29 08:50:04,702-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [358e5112] FINISH, SetVdsStatusVDSCommand, log id: 349a8bd4 >2018-05-29 08:50:04,710-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 358e5112, Job ID: 740b8bc2-144a-4ad7-b40b-5060ebe1dc40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:50:04,714-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ed20748, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:50:04,716-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [358e5112] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ed20748, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:50:04,727-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [553cdb0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:50:04,728-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [553cdb0e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:54:42,967-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [2480d1a] Attempting to update VMs/Templates Ovf. >2018-05-29 08:54:42,969-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 08:54:42,969-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 08:54:42,970-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:54:42,974-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 08:54:42,977-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 08:54:42,977-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 08:54:42,978-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 08:54:42,978-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 08:54:42,978-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 08:54:42,979-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [69ef064f] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 08:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2587f861] Autorecovering 2 hosts >2018-05-29 08:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2587f861] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [755ade9c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b750b0c >2018-05-29 08:55:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [755ade9c] FINISH, SetVdsStatusVDSCommand, log id: 5b750b0c >2018-05-29 08:55:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,056-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [755ade9c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 755ade9c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 08:55:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [755ade9c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 08:55:00,057-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [755ade9c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:55:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 08:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6249396 >2018-05-29 08:55:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] FINISH, SetVdsStatusVDSCommand, log id: 6249396 >2018-05-29 08:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4431c558 >2018-05-29 08:55:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [38f4e2ea] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4431c558 >2018-05-29 08:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 08:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [38f4e2ea] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 08:55:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [416dccd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 276d35f2 >2018-05-29 08:55:02,380-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [416dccd] FINISH, GetHardwareInfoVDSCommand, log id: 276d35f2 >2018-05-29 08:55:02,393-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [61c25662] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:55:02,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [69645bbe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b8c623 >2018-05-29 08:55:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [69645bbe] FINISH, GetHardwareInfoVDSCommand, log id: 1b8c623 >2018-05-29 08:55:02,402-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70570919] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:55:02,408-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [45db4d1b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:55:02,413-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [b2c5cf5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:55:02,416-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [45db4d1b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e4cdb5a >2018-05-29 08:55:02,418-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5910726] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:55:02,423-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [213eb676] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 08:55:02,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [213eb676] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b267189 >2018-05-29 08:55:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:55:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [45db4d1b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e4cdb5a >2018-05-29 08:55:02,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,794-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [45db4d1b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,802-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [45db4d1b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:55:02,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [45db4d1b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45db4d1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:55:02,811-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 08:55:02,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [213eb676] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b267189 >2018-05-29 08:55:02,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f2f1a17 >2018-05-29 08:55:02,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:55:02,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,816-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 08:55:02,825-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [213eb676] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 08:55:02,827-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [213eb676] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 08:55:02,830-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [213eb676] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 213eb676, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:55:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:55:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:55:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f2f1a17 >2018-05-29 08:55:02,908-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:55:02,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38113a51 >2018-05-29 08:55:02,910-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 08:55:02,921-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7271dfdd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:55:02,922-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7271dfdd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 570c319e >2018-05-29 08:55:02,924-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7271dfdd] FINISH, SetVdsStatusVDSCommand, log id: 570c319e >2018-05-29 08:55:02,954-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7271dfdd, Job ID: d4cda4bd-bf8d-4173-9c28-9d903cc302f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:55:02,958-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70570919, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:55:02,961-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7271dfdd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70570919, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:55:02,971-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [208a8419] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 08:55:02,972-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [208a8419] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:55:03,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 08:55:03,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 08:55:03,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38113a51 >2018-05-29 08:55:03,010-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 08:55:03,022-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42feb506] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:55:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42feb506] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 652373c9 >2018-05-29 08:55:03,026-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42feb506] FINISH, SetVdsStatusVDSCommand, log id: 652373c9 >2018-05-29 08:55:03,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42feb506, Job ID: 66bd4e3a-b5f5-4e6c-a142-525491bbed9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 08:55:03,036-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5910726, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 08:55:03,038-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42feb506] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5910726, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 08:55:03,048-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [51fdb5de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 08:55:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [51fdb5de] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:00:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [28399481] Autorecovering 2 hosts >2018-05-29 09:00:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [28399481] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:00:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3491c1ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 310561f5 >2018-05-29 09:00:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3491c1ef] FINISH, SetVdsStatusVDSCommand, log id: 310561f5 >2018-05-29 09:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3491c1ef] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:00:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3491c1ef] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:00:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:00:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:00:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [890723a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 244e932f >2018-05-29 09:00:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [890723a] FINISH, SetVdsStatusVDSCommand, log id: 244e932f >2018-05-29 09:00:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [890723a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5af2df07 >2018-05-29 09:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [890723a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5af2df07 >2018-05-29 09:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:00:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [890723a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:00:00,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [74e54887] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 48ca657a >2018-05-29 09:00:00,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [74e54887] FINISH, GetHardwareInfoVDSCommand, log id: 48ca657a >2018-05-29 09:00:00,624-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [626ee555] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:00:00,632-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4a9dd592] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:00:00,637-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d08f0b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:00:00,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d08f0b1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 201577c3 >2018-05-29 09:00:00,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4201690b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45189e37 >2018-05-29 09:00:00,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4201690b] FINISH, GetHardwareInfoVDSCommand, log id: 45189e37 >2018-05-29 09:00:00,725-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2544550d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:00:00,730-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [d38ddd8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:00:00,735-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [344b7562] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:00:00,741-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [344b7562] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ef65963 >2018-05-29 09:00:01,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:00:01,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d08f0b1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 201577c3 >2018-05-29 09:00:01,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,090-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,094-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,095-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d08f0b1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,097-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d08f0b1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:00:01,100-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d08f0b1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d08f0b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:00:01,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:00:01,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [344b7562] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ef65963 >2018-05-29 09:00:01,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68138bcb >2018-05-29 09:00:01,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:00:01,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:00:01,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [344b7562] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:00:01,124-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [344b7562] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:00:01,126-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [344b7562] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 344b7562, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:00:01,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:00:01,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:00:01,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68138bcb >2018-05-29 09:00:01,206-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:00:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4665467e >2018-05-29 09:00:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:00:01,220-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2e308d47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:00:01,221-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e308d47] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12cec1f >2018-05-29 09:00:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e308d47] FINISH, SetVdsStatusVDSCommand, log id: 12cec1f >2018-05-29 09:00:01,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e308d47, Job ID: 4d744a32-0b8f-431f-8bf3-35840cd5e8bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:00:01,262-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a9dd592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:00:01,264-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e308d47] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a9dd592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:00:01,275-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [25dd0677] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:00:01,276-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [25dd0677] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:00:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:00:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:00:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4665467e >2018-05-29 09:00:01,311-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:00:01,325-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4e7f29b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:00:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4e7f29b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f475cc9 >2018-05-29 09:00:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4e7f29b] FINISH, SetVdsStatusVDSCommand, log id: 7f475cc9 >2018-05-29 09:00:01,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e7f29b, Job ID: b7d92374-e15c-4a26-a317-366a9308775a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:00:01,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d38ddd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:00:01,340-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e7f29b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d38ddd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:00:01,351-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1d04c49d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:00:01,352-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1d04c49d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d7a46fe] Autorecovering 2 hosts >2018-05-29 09:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1d7a46fe] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ab9ad2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 180c0172 >2018-05-29 09:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ab9ad2e] FINISH, SetVdsStatusVDSCommand, log id: 180c0172 >2018-05-29 09:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ab9ad2e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ab9ad2e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b1d344] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 89bb85c >2018-05-29 09:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b1d344] FINISH, SetVdsStatusVDSCommand, log id: 89bb85c >2018-05-29 09:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b1d344] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2c098a5 >2018-05-29 09:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b1d344] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2c098a5 >2018-05-29 09:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:05:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b1d344] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:05:01,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7c93cb65] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 30ec9e2d >2018-05-29 09:05:01,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7c93cb65] FINISH, GetHardwareInfoVDSCommand, log id: 30ec9e2d >2018-05-29 09:05:01,906-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7960f0da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:05:01,915-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [56cbc57d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:05:01,920-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6356e22] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:05:01,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6356e22] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: c3e41c3 >2018-05-29 09:05:01,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [451d712d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4b0b554a >2018-05-29 09:05:01,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [451d712d] FINISH, GetHardwareInfoVDSCommand, log id: 4b0b554a >2018-05-29 09:05:01,996-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3e5b7e49] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:05:02,000-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [69862cb4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:05:02,005-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6c969340] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:05:02,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6c969340] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10c08071 >2018-05-29 09:05:02,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:05:02,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6356e22] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: c3e41c3 >2018-05-29 09:05:02,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6356e22] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6356e22] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:05:02,339-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6356e22] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6356e22, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:05:02,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bf9a434 >2018-05-29 09:05:02,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:05:02,376-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:05:02,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6c969340] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10c08071 >2018-05-29 09:05:02,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,381-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:05:02,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6c969340] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:05:02,392-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6c969340] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:05:02,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6c969340] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c969340, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:05:02,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:05:02,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:05:02,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bf9a434 >2018-05-29 09:05:02,447-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:05:02,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 490e8fdc >2018-05-29 09:05:02,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:05:02,461-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [942bc2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:05:02,462-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [942bc2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8fe8cfb >2018-05-29 09:05:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [942bc2b] FINISH, SetVdsStatusVDSCommand, log id: 8fe8cfb >2018-05-29 09:05:02,497-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 942bc2b, Job ID: fc59ec3d-5997-468e-a2dc-0acb7832aadb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:05:02,502-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 56cbc57d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:05:02,504-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [942bc2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 56cbc57d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:05:02,515-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5b57c60e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:05:02,516-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5b57c60e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 490e8fdc >2018-05-29 09:05:02,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:05:02,562-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [22d56af5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:05:02,563-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22d56af5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25a63ea1 >2018-05-29 09:05:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [22d56af5] FINISH, SetVdsStatusVDSCommand, log id: 25a63ea1 >2018-05-29 09:05:02,598-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22d56af5, Job ID: 00601919-57bf-4a9c-b700-d141670821d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:05:02,604-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69862cb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:05:02,608-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [22d56af5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69862cb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:05:02,619-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b3d6efb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:05:02,620-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b3d6efb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [17a5d615] Autorecovering 2 hosts >2018-05-29 09:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [17a5d615] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5de5e5e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 229d371f >2018-05-29 09:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5de5e5e2] FINISH, SetVdsStatusVDSCommand, log id: 229d371f >2018-05-29 09:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5de5e5e2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5de5e5e2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60fccb83 >2018-05-29 09:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] FINISH, SetVdsStatusVDSCommand, log id: 60fccb83 >2018-05-29 09:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5194777f >2018-05-29 09:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2ca9735e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5194777f >2018-05-29 09:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2ca9735e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:10:00,277-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [358f6c7b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76af16be >2018-05-29 09:10:00,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [358f6c7b] FINISH, GetHardwareInfoVDSCommand, log id: 76af16be >2018-05-29 09:10:00,293-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4490b2a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:10:00,301-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2bb581e1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:10:00,306-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d2d2c5a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:10:00,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d2d2c5a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 643d63b8 >2018-05-29 09:10:00,696-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:10:00,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2d2d2c5a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 643d63b8 >2018-05-29 09:10:00,700-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:00,701-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:00,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:00,706-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:00,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:00,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2d2d2c5a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:00,713-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2d2d2c5a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:10:00,716-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d2d2c5a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d2d2c5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:10:00,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e98eded >2018-05-29 09:10:00,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:10:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:10:00,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:10:00,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:10:00,821-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:10:00,821-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e98eded >2018-05-29 09:10:00,821-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:10:00,835-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2defbb28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:10:00,836-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2defbb28] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc84ca6 >2018-05-29 09:10:00,838-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2defbb28] FINISH, SetVdsStatusVDSCommand, log id: 2fc84ca6 >2018-05-29 09:10:00,870-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2defbb28, Job ID: b23d790e-5a0d-4783-b602-14143c0c21a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:10:00,875-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bb581e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:10:00,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2defbb28] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bb581e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:10:00,888-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2693cfc0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:10:00,889-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2693cfc0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:10:03,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12d8217f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 31a31ef8 >2018-05-29 09:10:03,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [12d8217f] FINISH, GetHardwareInfoVDSCommand, log id: 31a31ef8 >2018-05-29 09:10:03,146-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [75883f97] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:10:03,150-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [42c69a24] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:10:03,156-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3bea2208] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:10:03,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3bea2208] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 501b2d38 >2018-05-29 09:10:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:10:03,541-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3bea2208] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 501b2d38 >2018-05-29 09:10:03,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:03,546-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:03,550-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:03,551-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:03,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:10:03,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3bea2208] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:10:03,558-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3bea2208] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:10:03,561-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3bea2208] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bea2208, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:10:03,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13814250 >2018-05-29 09:10:03,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:10:03,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:10:03,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:10:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:10:03,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:10:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13814250 >2018-05-29 09:10:03,664-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:10:03,677-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [67b57bd0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:10:03,678-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [67b57bd0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c31b080 >2018-05-29 09:10:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [67b57bd0] FINISH, SetVdsStatusVDSCommand, log id: 7c31b080 >2018-05-29 09:10:03,686-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 67b57bd0, Job ID: 525f86b5-1d1c-4b19-b9f3-dfc244616085, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:10:03,691-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 42c69a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:10:03,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [67b57bd0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 42c69a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:10:03,703-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [45ece9bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:10:03,704-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [45ece9bc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1bbc1178] Autorecovering 2 hosts >2018-05-29 09:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1bbc1178] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57cd2c14] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bff1161 >2018-05-29 09:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57cd2c14] FINISH, SetVdsStatusVDSCommand, log id: 3bff1161 >2018-05-29 09:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [57cd2c14] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:15:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [57cd2c14] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:15:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [c545014] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f552e8 >2018-05-29 09:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [c545014] FINISH, SetVdsStatusVDSCommand, log id: 1f552e8 >2018-05-29 09:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [c545014] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43336618 >2018-05-29 09:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [c545014] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43336618 >2018-05-29 09:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:15:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [c545014] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:15:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2595cb7e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 58e5708f >2018-05-29 09:15:01,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2595cb7e] FINISH, GetHardwareInfoVDSCommand, log id: 58e5708f >2018-05-29 09:15:01,355-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [62af264] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:15:01,363-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [28fd1222] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:15:01,369-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [73274eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:15:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [73274eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ea0adcb >2018-05-29 09:15:01,578-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [186fb4aa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7940be1d >2018-05-29 09:15:01,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [186fb4aa] FINISH, GetHardwareInfoVDSCommand, log id: 7940be1d >2018-05-29 09:15:01,592-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1b77705f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:15:01,597-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [346eb320] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:15:01,602-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69a48446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:15:01,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69a48446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 52916e0e >2018-05-29 09:15:01,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:15:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [73274eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ea0adcb >2018-05-29 09:15:01,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,773-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:01,777-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:01,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,782-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [73274eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:01,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [73274eb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:15:01,787-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [73274eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73274eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:15:01,794-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 635c270a >2018-05-29 09:15:01,794-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:15:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:15:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:15:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 635c270a >2018-05-29 09:15:01,892-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:15:01,906-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [575d9383] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:15:01,907-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [575d9383] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6083d0c8 >2018-05-29 09:15:01,910-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [575d9383] FINISH, SetVdsStatusVDSCommand, log id: 6083d0c8 >2018-05-29 09:15:01,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 575d9383, Job ID: 78282384-13f0-476e-a6c8-771f2f9f0a15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:15:01,946-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 28fd1222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:15:01,949-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [575d9383] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 28fd1222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:15:01,959-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [33b422d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:15:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [33b422d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:15:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:15:01,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69a48446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 52916e0e >2018-05-29 09:15:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:01,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:15:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69a48446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:15:02,000-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69a48446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:15:02,003-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69a48446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69a48446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:15:02,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74a34a9f >2018-05-29 09:15:02,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:15:02,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:15:02,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:15:02,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74a34a9f >2018-05-29 09:15:02,101-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:15:02,114-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3763799] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:15:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3763799] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2774127d >2018-05-29 09:15:02,117-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3763799] FINISH, SetVdsStatusVDSCommand, log id: 2774127d >2018-05-29 09:15:02,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3763799, Job ID: d31a2453-6895-431c-a443-0a6222763b5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:15:02,127-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 346eb320, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:15:02,129-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3763799] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 346eb320, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:15:02,140-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6dc7f21c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:15:02,141-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6dc7f21c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [320f0eef] Autorecovering 2 hosts >2018-05-29 09:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [320f0eef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:20:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:20:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5645e60d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9bb97b1 >2018-05-29 09:20:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5645e60d] FINISH, SetVdsStatusVDSCommand, log id: 9bb97b1 >2018-05-29 09:20:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5645e60d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:20:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5645e60d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:20:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50ded7 >2018-05-29 09:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] FINISH, SetVdsStatusVDSCommand, log id: 50ded7 >2018-05-29 09:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ee562d0 >2018-05-29 09:20:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [5782d0e9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ee562d0 >2018-05-29 09:20:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:20:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5782d0e9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:20:02,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [629dce32] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6e30584a >2018-05-29 09:20:02,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [629dce32] FINISH, GetHardwareInfoVDSCommand, log id: 6e30584a >2018-05-29 09:20:02,668-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [77ae512c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:20:02,676-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [47b2f001] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:20:02,681-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [77934311] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:20:02,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [77934311] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f0ee768 >2018-05-29 09:20:02,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [771be597] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3758af3d >2018-05-29 09:20:02,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [771be597] FINISH, GetHardwareInfoVDSCommand, log id: 3758af3d >2018-05-29 09:20:02,820-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [68114cb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:20:02,825-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [19338e5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:20:02,830-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [53282368] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:20:02,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [53282368] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2e2189ce >2018-05-29 09:20:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:20:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [53282368] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2e2189ce >2018-05-29 09:20:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:03,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:03,235-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:03,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [53282368] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:03,243-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [53282368] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:20:03,246-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [53282368] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53282368, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:20:03,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 71efa136 >2018-05-29 09:20:03,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:20:03,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:20:03,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:20:03,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 71efa136 >2018-05-29 09:20:03,352-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:20:03,366-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [73c14583] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:20:03,368-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c14583] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b09167b >2018-05-29 09:20:03,370-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c14583] FINISH, SetVdsStatusVDSCommand, log id: 6b09167b >2018-05-29 09:20:03,401-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73c14583, Job ID: 79172dde-18f2-42ec-b082-ad02daf7fef4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:20:03,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19338e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:20:03,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73c14583] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19338e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:20:03,420-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7a229726] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:20:03,421-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a229726] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:20:04,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:20:04,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [77934311] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f0ee768 >2018-05-29 09:20:04,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:04,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:04,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:04,072-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:04,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:20:04,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [77934311] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:20:04,080-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [77934311] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:20:04,083-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [77934311] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77934311, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:20:04,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50d935fb >2018-05-29 09:20:04,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:20:04,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:20:04,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:20:04,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50d935fb >2018-05-29 09:20:04,189-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:20:04,203-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [38404fb6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:20:04,204-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [38404fb6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5155feae >2018-05-29 09:20:04,207-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [38404fb6] FINISH, SetVdsStatusVDSCommand, log id: 5155feae >2018-05-29 09:20:04,242-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38404fb6, Job ID: 6a3a4296-f5e5-4060-81b6-dca7b10b0b6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:20:04,247-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47b2f001, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:20:04,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [38404fb6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47b2f001, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:20:04,261-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3ae3e2a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:20:04,262-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3ae3e2a2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75e8a26b] Autorecovering 2 hosts >2018-05-29 09:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75e8a26b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [568af21] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22844149 >2018-05-29 09:25:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [568af21] FINISH, SetVdsStatusVDSCommand, log id: 22844149 >2018-05-29 09:25:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [568af21] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:25:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [568af21] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:25:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:25:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:25:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17e59f53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c7346b >2018-05-29 09:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17e59f53] FINISH, SetVdsStatusVDSCommand, log id: 3c7346b >2018-05-29 09:25:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17e59f53] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36511954 >2018-05-29 09:25:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17e59f53] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36511954 >2018-05-29 09:25:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:25:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17e59f53] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:25:01,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [41fd8d39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 590d2937 >2018-05-29 09:25:01,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [41fd8d39] FINISH, GetHardwareInfoVDSCommand, log id: 590d2937 >2018-05-29 09:25:01,175-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [96575fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:25:01,184-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [41c01cba] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:25:01,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [449e5c22] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:25:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [449e5c22] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ba862fe >2018-05-29 09:25:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:25:01,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [449e5c22] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ba862fe >2018-05-29 09:25:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:01,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:01,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:01,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [449e5c22] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:01,584-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [449e5c22] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:25:01,586-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [449e5c22] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 449e5c22, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:25:01,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1083206c >2018-05-29 09:25:01,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:25:01,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:25:01,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:25:01,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1083206c >2018-05-29 09:25:01,686-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:25:01,699-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [76cdd52e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:25:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [76cdd52e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b622ebd >2018-05-29 09:25:01,703-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [76cdd52e] FINISH, SetVdsStatusVDSCommand, log id: b622ebd >2018-05-29 09:25:01,708-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 76cdd52e, Job ID: aa5a145d-2be2-4b25-a86d-b240688f6839, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:25:01,712-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41c01cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:25:01,715-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [76cdd52e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41c01cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:25:01,725-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [9933632] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:25:01,726-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [9933632] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:25:01,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2f709f64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26cd97c3 >2018-05-29 09:25:01,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2f709f64] FINISH, GetHardwareInfoVDSCommand, log id: 26cd97c3 >2018-05-29 09:25:01,871-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [415ba984] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:25:01,875-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [33fb95eb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:25:01,881-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1a1eab33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:25:01,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1a1eab33] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69ace64f >2018-05-29 09:25:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:25:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1a1eab33] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69ace64f >2018-05-29 09:25:02,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:02,264-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:02,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:02,269-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:02,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:25:02,274-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1a1eab33] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:25:02,276-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1a1eab33] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:25:02,279-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1a1eab33] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a1eab33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:25:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 423b13ff >2018-05-29 09:25:02,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:25:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:25:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:25:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 423b13ff >2018-05-29 09:25:02,378-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:25:02,391-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [31e19f1d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:25:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [31e19f1d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c34111a >2018-05-29 09:25:02,394-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [31e19f1d] FINISH, SetVdsStatusVDSCommand, log id: 2c34111a >2018-05-29 09:25:02,400-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31e19f1d, Job ID: c94737c2-62fa-4bd0-866f-468788afa215, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:25:02,405-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33fb95eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:25:02,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [31e19f1d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33fb95eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:25:02,417-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [15bb2508] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:25:02,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [15bb2508] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5b6c0a1c] Autorecovering 2 hosts >2018-05-29 09:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5b6c0a1c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [770eec3a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2072b17d >2018-05-29 09:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [770eec3a] FINISH, SetVdsStatusVDSCommand, log id: 2072b17d >2018-05-29 09:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [770eec3a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:30:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [770eec3a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [bf11c23] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13642be7 >2018-05-29 09:30:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [bf11c23] FINISH, SetVdsStatusVDSCommand, log id: 13642be7 >2018-05-29 09:30:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [bf11c23] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: ca2116b >2018-05-29 09:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [bf11c23] FINISH, SetHaMaintenanceModeVDSCommand, log id: ca2116b >2018-05-29 09:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [bf11c23] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:30:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [41c4890] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c0d57a7 >2018-05-29 09:30:02,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [41c4890] FINISH, GetHardwareInfoVDSCommand, log id: 7c0d57a7 >2018-05-29 09:30:02,480-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3b206289] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:30:02,489-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [379dc020] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:30:02,494-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [19e64136] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:30:02,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [19e64136] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7861be6 >2018-05-29 09:30:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:30:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [19e64136] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7861be6 >2018-05-29 09:30:02,890-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:02,890-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:02,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:02,895-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:02,899-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [19e64136] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:02,902-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [19e64136] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:30:02,904-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [19e64136] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 19e64136, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:30:02,910-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1abed62b >2018-05-29 09:30:02,910-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:30:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [182520c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4cafb244 >2018-05-29 09:30:02,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [182520c] FINISH, GetHardwareInfoVDSCommand, log id: 4cafb244 >2018-05-29 09:30:03,001-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [71acd3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:30:03,005-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [24e74bf9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:30:03,010-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [64f235d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:30:03,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:30:03,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:30:03,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1abed62b >2018-05-29 09:30:03,013-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:30:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [64f235d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 497a0f6f >2018-05-29 09:30:03,026-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [258ad0a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:30:03,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [258ad0a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 207bc276 >2018-05-29 09:30:03,029-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [258ad0a3] FINISH, SetVdsStatusVDSCommand, log id: 207bc276 >2018-05-29 09:30:03,061-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 258ad0a3, Job ID: fd602033-6a7b-46c3-9bc2-4385645d6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:30:03,066-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 379dc020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:30:03,068-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [258ad0a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 379dc020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:30:03,079-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1e66b955] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:30:03,080-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1e66b955] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:30:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:30:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [64f235d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 497a0f6f >2018-05-29 09:30:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:30:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [64f235d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:30:03,419-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [64f235d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:30:03,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [64f235d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 64f235d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:30:03,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e3f1761 >2018-05-29 09:30:03,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:30:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:30:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:30:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: e3f1761 >2018-05-29 09:30:03,528-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:30:03,541-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [e7bba15] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:30:03,543-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [e7bba15] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a6e992 >2018-05-29 09:30:03,545-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [e7bba15] FINISH, SetVdsStatusVDSCommand, log id: 17a6e992 >2018-05-29 09:30:03,550-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e7bba15, Job ID: 494fea00-a535-48e6-bfcd-7877a40933a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:30:03,554-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24e74bf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:30:03,556-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [e7bba15] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24e74bf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:30:03,567-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6c2919a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:30:03,568-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6c2919a7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5fffab58] Autorecovering 2 hosts >2018-05-29 09:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [5fffab58] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:35:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:35:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4ea4a581] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 448e3ad1 >2018-05-29 09:35:00,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4ea4a581] FINISH, SetVdsStatusVDSCommand, log id: 448e3ad1 >2018-05-29 09:35:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4ea4a581] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:35:00,057-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4ea4a581] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:35:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: def8815 >2018-05-29 09:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] FINISH, SetVdsStatusVDSCommand, log id: def8815 >2018-05-29 09:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6c1dfe77 >2018-05-29 09:35:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4fe8a080] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6c1dfe77 >2018-05-29 09:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:35:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4fe8a080] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:35:00,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6024a089] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8b80e31 >2018-05-29 09:35:00,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6024a089] FINISH, GetHardwareInfoVDSCommand, log id: 8b80e31 >2018-05-29 09:35:00,871-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [50beb0a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:35:00,880-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [c0ab2b3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:35:00,885-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [15b350f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:35:00,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [15b350f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5253a57b >2018-05-29 09:35:01,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77c6e3a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23325434 >2018-05-29 09:35:01,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77c6e3a5] FINISH, GetHardwareInfoVDSCommand, log id: 23325434 >2018-05-29 09:35:01,123-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [a54eae6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:35:01,127-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [7a7bdf30] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:35:01,133-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5f2a13ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:35:01,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5f2a13ad] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 76d6d10a >2018-05-29 09:35:01,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:35:01,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5f2a13ad] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 76d6d10a >2018-05-29 09:35:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:01,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:01,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:01,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:01,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:01,531-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5f2a13ad] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:01,534-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5f2a13ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:35:01,537-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5f2a13ad] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f2a13ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:35:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 522aeeba >2018-05-29 09:35:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:35:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:35:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:35:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 522aeeba >2018-05-29 09:35:01,639-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:35:01,653-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2405b1fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:35:01,655-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2405b1fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 915e5b6 >2018-05-29 09:35:01,657-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2405b1fd] FINISH, SetVdsStatusVDSCommand, log id: 915e5b6 >2018-05-29 09:35:01,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2405b1fd, Job ID: 3868a1b6-c557-43d4-bfc6-2ecd28c067c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:35:01,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a7bdf30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:35:01,696-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2405b1fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a7bdf30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:35:01,707-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3cbeaaad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:35:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3cbeaaad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:35:02,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:35:02,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [15b350f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5253a57b >2018-05-29 09:35:02,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:02,262-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:02,266-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:02,267-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:02,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:35:02,271-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [15b350f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:35:02,275-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [15b350f5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:35:02,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15b350f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 15b350f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:35:02,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 411974a6 >2018-05-29 09:35:02,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:35:02,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:35:02,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:35:02,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 411974a6 >2018-05-29 09:35:02,376-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:35:02,389-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [16112223] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:35:02,390-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16112223] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ee3c343 >2018-05-29 09:35:02,393-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16112223] FINISH, SetVdsStatusVDSCommand, log id: 7ee3c343 >2018-05-29 09:35:02,398-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16112223, Job ID: b7b21f36-938f-48f4-983a-f300e0725b14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:35:02,403-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c0ab2b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:35:02,405-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16112223] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c0ab2b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:35:02,416-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5f214e3c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:35:02,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5f214e3c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [108ac6b4] Autorecovering 2 hosts >2018-05-29 09:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [108ac6b4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e6d7ebf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6247bcd >2018-05-29 09:40:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e6d7ebf] FINISH, SetVdsStatusVDSCommand, log id: 6247bcd >2018-05-29 09:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e6d7ebf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:40:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6e6d7ebf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 678c492c >2018-05-29 09:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] FINISH, SetVdsStatusVDSCommand, log id: 678c492c >2018-05-29 09:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 706b3e30 >2018-05-29 09:40:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5ad589d1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 706b3e30 >2018-05-29 09:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:40:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5ad589d1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:40:02,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fbbc8df] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f6e1b32 >2018-05-29 09:40:02,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fbbc8df] FINISH, GetHardwareInfoVDSCommand, log id: 6f6e1b32 >2018-05-29 09:40:02,348-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [29cf889] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:40:02,356-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d982c93] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:40:02,362-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41332d1f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:40:02,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41332d1f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1f571aaf >2018-05-29 09:40:02,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:40:02,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41332d1f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1f571aaf >2018-05-29 09:40:02,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:02,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:02,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:02,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:02,764-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41332d1f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:02,766-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41332d1f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:40:02,769-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41332d1f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41332d1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:40:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5bd38a88 >2018-05-29 09:40:02,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:40:02,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:40:02,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:40:02,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5bd38a88 >2018-05-29 09:40:02,873-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:40:02,887-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [32e5856f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:40:02,888-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32e5856f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 392f1cea >2018-05-29 09:40:02,890-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32e5856f] FINISH, SetVdsStatusVDSCommand, log id: 392f1cea >2018-05-29 09:40:02,922-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32e5856f, Job ID: 04838009-cc03-4cf0-92b7-72994a9ea26c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:40:02,927-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d982c93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:40:02,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32e5856f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d982c93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:40:02,941-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6b79fcb4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:40:02,942-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6b79fcb4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:40:03,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dc871a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c07c03c >2018-05-29 09:40:03,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2dc871a] FINISH, GetHardwareInfoVDSCommand, log id: 7c07c03c >2018-05-29 09:40:03,157-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [b882535] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:40:03,161-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4ebd025b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:40:03,167-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [b31a672] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:40:03,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [b31a672] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 220324f6 >2018-05-29 09:40:03,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:40:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [b31a672] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 220324f6 >2018-05-29 09:40:03,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:03,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:03,560-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:03,561-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:03,565-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:40:03,565-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [b31a672] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:40:03,568-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [b31a672] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:40:03,571-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [b31a672] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: b31a672, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:40:03,574-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d35bcf >2018-05-29 09:40:03,574-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:40:03,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:40:03,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:40:03,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d35bcf >2018-05-29 09:40:03,672-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:40:03,686-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4d671fde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:40:03,687-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4d671fde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 212824ad >2018-05-29 09:40:03,689-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4d671fde] FINISH, SetVdsStatusVDSCommand, log id: 212824ad >2018-05-29 09:40:03,695-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d671fde, Job ID: 4c49a043-8360-432c-9b00-5f23136c57fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:40:03,700-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4ebd025b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:40:03,702-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4d671fde] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4ebd025b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:40:03,712-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [648e83cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:40:03,713-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [648e83cd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [66c0b3fc] Autorecovering 2 hosts >2018-05-29 09:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [66c0b3fc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:45:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [509052e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dc42f64 >2018-05-29 09:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [509052e1] FINISH, SetVdsStatusVDSCommand, log id: 3dc42f64 >2018-05-29 09:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [509052e1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:45:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [509052e1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:45:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:45:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:45:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40036a4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3439c1b2 >2018-05-29 09:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [40036a4e] FINISH, SetVdsStatusVDSCommand, log id: 3439c1b2 >2018-05-29 09:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [40036a4e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51f04703 >2018-05-29 09:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [40036a4e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51f04703 >2018-05-29 09:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [40036a4e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:45:00,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [679a22e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 893c3c3 >2018-05-29 09:45:00,514-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [679a22e4] FINISH, GetHardwareInfoVDSCommand, log id: 893c3c3 >2018-05-29 09:45:00,528-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2bd68be4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:45:00,537-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [17b37e34] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:45:00,542-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3fe9e097] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:45:00,553-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3fe9e097] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b16df26 >2018-05-29 09:45:00,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:45:00,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3fe9e097] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b16df26 >2018-05-29 09:45:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:00,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:00,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:00,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3fe9e097] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:00,948-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3fe9e097] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:45:00,951-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3fe9e097] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3fe9e097, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:45:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b888fae >2018-05-29 09:45:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:45:01,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:45:01,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:45:01,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b888fae >2018-05-29 09:45:01,056-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:45:01,070-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [30a614b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:45:01,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [30a614b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c333300 >2018-05-29 09:45:01,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [30a614b] FINISH, SetVdsStatusVDSCommand, log id: 5c333300 >2018-05-29 09:45:01,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 30a614b, Job ID: 7eb63609-24d7-466e-a1c1-fc34aef075fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:45:01,110-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17b37e34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:45:01,112-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [30a614b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17b37e34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:45:01,123-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [29c6e35a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:45:01,124-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [29c6e35a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:45:01,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3f01b117] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38e51678 >2018-05-29 09:45:01,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3f01b117] FINISH, GetHardwareInfoVDSCommand, log id: 38e51678 >2018-05-29 09:45:01,485-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [598b3def] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:45:01,490-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [288d0f25] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:45:01,496-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e154fed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:45:01,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e154fed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6311153c >2018-05-29 09:45:01,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:45:01,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e154fed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6311153c >2018-05-29 09:45:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:01,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:01,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:01,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:01,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:45:01,883-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e154fed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:45:01,886-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e154fed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:45:01,889-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e154fed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e154fed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:45:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ec36dbe >2018-05-29 09:45:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:45:01,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:45:01,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:45:01,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ec36dbe >2018-05-29 09:45:01,988-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:45:02,001-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7df81419] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:45:02,002-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7df81419] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e81a704 >2018-05-29 09:45:02,004-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7df81419] FINISH, SetVdsStatusVDSCommand, log id: 5e81a704 >2018-05-29 09:45:02,010-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7df81419, Job ID: 2e77edc1-6f5c-4870-8474-17444dea182f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:45:02,015-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 288d0f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:45:02,017-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7df81419] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 288d0f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:45:02,027-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2804112d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:45:02,028-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2804112d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4a2cf1e] Autorecovering 2 hosts >2018-05-29 09:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4a2cf1e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [426f77a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14e3e2c0 >2018-05-29 09:50:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [426f77a7] FINISH, SetVdsStatusVDSCommand, log id: 14e3e2c0 >2018-05-29 09:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [426f77a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:50:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [426f77a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:50:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53d64cca >2018-05-29 09:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] FINISH, SetVdsStatusVDSCommand, log id: 53d64cca >2018-05-29 09:50:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f82aaf5 >2018-05-29 09:50:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [25b3d09f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f82aaf5 >2018-05-29 09:50:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:50:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [25b3d09f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:50:01,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ce0f7ce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78fa9a4a >2018-05-29 09:50:01,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ce0f7ce] FINISH, GetHardwareInfoVDSCommand, log id: 78fa9a4a >2018-05-29 09:50:01,727-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7285f44c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:50:01,736-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52b1a238] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:50:01,741-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5431bb69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:50:01,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5431bb69] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5056720d >2018-05-29 09:50:02,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:50:02,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5431bb69] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5056720d >2018-05-29 09:50:02,136-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:02,137-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:02,140-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:02,141-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:02,145-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:02,146-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5431bb69] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:02,149-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5431bb69] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:50:02,152-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5431bb69] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5431bb69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:50:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50ee785c >2018-05-29 09:50:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:50:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:50:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:50:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50ee785c >2018-05-29 09:50:02,259-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:50:02,272-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [612d0728] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:50:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [612d0728] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3955d8ad >2018-05-29 09:50:02,276-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [612d0728] FINISH, SetVdsStatusVDSCommand, log id: 3955d8ad >2018-05-29 09:50:02,308-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 612d0728, Job ID: 714fc5ff-d205-4571-9b51-7772ff2d933a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:50:02,314-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52b1a238, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:50:02,316-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [612d0728] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52b1a238, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:50:02,328-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3980f41a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:50:02,329-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3980f41a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:50:02,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4f4b683e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 49984e73 >2018-05-29 09:50:02,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4f4b683e] FINISH, GetHardwareInfoVDSCommand, log id: 49984e73 >2018-05-29 09:50:02,767-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [738e0835] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:50:02,772-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [15e35023] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:50:02,777-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [290e55c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:50:02,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [290e55c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25b06092 >2018-05-29 09:50:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:50:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [290e55c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25b06092 >2018-05-29 09:50:03,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:03,159-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:03,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:03,164-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:03,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:50:03,168-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [290e55c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:50:03,172-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [290e55c4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:50:03,174-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [290e55c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 290e55c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:50:03,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4516e62d >2018-05-29 09:50:03,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:50:03,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:50:03,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:50:03,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:50:03,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:50:03,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4516e62d >2018-05-29 09:50:03,275-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:50:03,288-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [94f160d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:50:03,289-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [94f160d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3fdbdafa >2018-05-29 09:50:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [94f160d] FINISH, SetVdsStatusVDSCommand, log id: 3fdbdafa >2018-05-29 09:50:03,297-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 94f160d, Job ID: c76ec90f-134a-4a9d-aa4d-e4167b4b8d48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:50:03,301-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15e35023, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:50:03,303-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [94f160d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15e35023, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:50:03,313-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [49374cc8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:50:03,314-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [49374cc8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:54:42,979-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [3a649785] Attempting to update VMs/Templates Ovf. >2018-05-29 09:54:42,982-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 09:54:42,982-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 09:54:42,983-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:54:42,987-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 09:54:42,990-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 09:54:42,990-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 09:54:42,990-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 09:54:42,990-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 09:54:42,991-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 09:54:42,991-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [75d793cd] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 09:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [a8fd26e] Autorecovering 2 hosts >2018-05-29 09:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [a8fd26e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:55:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [161ac0c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53a532a >2018-05-29 09:55:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [161ac0c6] FINISH, SetVdsStatusVDSCommand, log id: 53a532a >2018-05-29 09:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [161ac0c6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 09:55:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [161ac0c6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 09:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2740f644] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 703b1f74 >2018-05-29 09:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2740f644] FINISH, SetVdsStatusVDSCommand, log id: 703b1f74 >2018-05-29 09:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2740f644] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 10ea5340 >2018-05-29 09:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2740f644] FINISH, SetHaMaintenanceModeVDSCommand, log id: 10ea5340 >2018-05-29 09:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 09:55:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2740f644] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 09:55:01,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d97d98] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5d09e951 >2018-05-29 09:55:01,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d97d98] FINISH, GetHardwareInfoVDSCommand, log id: 5d09e951 >2018-05-29 09:55:01,026-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [40135cb2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:55:01,035-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1007512b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:55:01,040-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [68a86e5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:55:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [68a86e5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 355b05d1 >2018-05-29 09:55:02,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:55:02,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [68a86e5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 355b05d1 >2018-05-29 09:55:02,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:02,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:02,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:02,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:02,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:02,442-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [68a86e5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:02,445-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [68a86e5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:55:02,447-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [68a86e5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68a86e5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:55:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a89c17b >2018-05-29 09:55:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:55:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:55:02,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:55:02,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 09:55:02,551-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:55:02,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a89c17b >2018-05-29 09:55:02,552-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:55:02,564-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [36ee077b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:55:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [36ee077b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a2cf2bd >2018-05-29 09:55:02,568-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [36ee077b] FINISH, SetVdsStatusVDSCommand, log id: 5a2cf2bd >2018-05-29 09:55:02,598-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36ee077b, Job ID: 3d3d6690-93f3-49a8-b603-5e717fe3730f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:55:02,603-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1007512b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:55:02,605-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [36ee077b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1007512b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:55:02,618-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [7d26fe1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 09:55:02,619-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7d26fe1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 09:55:03,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5c8c1231] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 138a81e0 >2018-05-29 09:55:03,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5c8c1231] FINISH, GetHardwareInfoVDSCommand, log id: 138a81e0 >2018-05-29 09:55:03,087-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [42ec113e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:55:03,091-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1462f5f0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:55:03,097-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [737120b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 09:55:03,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [737120b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4642d425 >2018-05-29 09:55:03,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 09:55:03,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [737120b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4642d425 >2018-05-29 09:55:03,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:03,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:03,485-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 09:55:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [737120b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 09:55:03,493-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [737120b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 09:55:03,496-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [737120b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 737120b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 09:55:03,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ba1ffef >2018-05-29 09:55:03,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:55:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 09:55:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 09:55:03,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ba1ffef >2018-05-29 09:55:03,594-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 09:55:03,607-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [d48853] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:55:03,608-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [d48853] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1711f144 >2018-05-29 09:55:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [d48853] FINISH, SetVdsStatusVDSCommand, log id: 1711f144 >2018-05-29 09:55:03,616-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: d48853, Job ID: fe434561-98b0-4494-b38e-ac8a468229f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 09:55:03,621-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1462f5f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 09:55:03,623-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [d48853] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1462f5f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 09:55:03,634-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1a16ac23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 09:55:03,635-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1a16ac23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:00:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50386b8] Autorecovering 2 hosts >2018-05-29 10:00:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [50386b8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:00:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:00:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:00:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [26ce2045] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 125309d2 >2018-05-29 10:00:01,918-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [26ce2045] FINISH, SetVdsStatusVDSCommand, log id: 125309d2 >2018-05-29 10:00:01,919-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:01,921-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [26ce2045] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:00:01,921-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [26ce2045] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:00:01,927-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:00:01,933-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:00:01,933-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:01,933-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:01,935-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [481b2b59] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8dab266 >2018-05-29 10:00:01,937-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [481b2b59] FINISH, SetVdsStatusVDSCommand, log id: 8dab266 >2018-05-29 10:00:01,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [481b2b59] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 729c56b7 >2018-05-29 10:00:01,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [481b2b59] FINISH, SetHaMaintenanceModeVDSCommand, log id: 729c56b7 >2018-05-29 10:00:01,942-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:00:01,943-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [481b2b59] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:00:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [34c79e2c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76a2eded >2018-05-29 10:00:03,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [34c79e2c] FINISH, GetHardwareInfoVDSCommand, log id: 76a2eded >2018-05-29 10:00:03,242-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2de44eb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:00:03,251-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [742b69f4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:00:03,256-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51949ee1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:00:03,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51949ee1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5df3df0d >2018-05-29 10:00:03,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:00:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51949ee1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5df3df0d >2018-05-29 10:00:03,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:03,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:03,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:03,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:03,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51949ee1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:03,660-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51949ee1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:00:03,662-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51949ee1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51949ee1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:00:03,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 761d2cda >2018-05-29 10:00:03,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:00:03,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:00:03,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:00:03,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 761d2cda >2018-05-29 10:00:03,767-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:00:03,780-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5d56d8a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:00:03,782-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5d56d8a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f3ce39d >2018-05-29 10:00:03,784-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5d56d8a3] FINISH, SetVdsStatusVDSCommand, log id: 2f3ce39d >2018-05-29 10:00:03,816-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d56d8a3, Job ID: b687a74d-4524-4444-aa31-ce7cf1535463, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:00:03,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 742b69f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:00:03,824-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d56d8a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 742b69f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:00:03,836-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [34acc635] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:00:03,837-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [34acc635] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:00:04,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [65b9de25] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: c96c8f8 >2018-05-29 10:00:04,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [65b9de25] FINISH, GetHardwareInfoVDSCommand, log id: c96c8f8 >2018-05-29 10:00:04,395-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5e902ba2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:00:04,404-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6c0bf229] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:00:04,409-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [79d03b8d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:00:04,420-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [79d03b8d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2324a1e7 >2018-05-29 10:00:04,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:00:04,800-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [79d03b8d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2324a1e7 >2018-05-29 10:00:04,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:04,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:04,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:04,810-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:04,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:00:04,814-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [79d03b8d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:00:04,817-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [79d03b8d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:00:04,820-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79d03b8d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 79d03b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:00:04,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 755701f3 >2018-05-29 10:00:04,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:00:04,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:00:04,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:00:04,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 755701f3 >2018-05-29 10:00:04,927-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:00:04,941-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3f39258] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:00:04,942-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3f39258] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2456ddf7 >2018-05-29 10:00:04,945-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3f39258] FINISH, SetVdsStatusVDSCommand, log id: 2456ddf7 >2018-05-29 10:00:04,977-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f39258, Job ID: 75a6e45c-41dd-4082-a8d7-06973e73f8c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:00:04,982-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6c0bf229, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:00:04,984-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3f39258] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6c0bf229, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:00:04,996-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1fc5d342] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:00:04,997-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1fc5d342] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79e3fa26] Autorecovering 2 hosts >2018-05-29 10:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79e3fa26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7c6f662e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b898196 >2018-05-29 10:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7c6f662e] FINISH, SetVdsStatusVDSCommand, log id: 3b898196 >2018-05-29 10:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7c6f662e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:05:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7c6f662e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [54b27274] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6767eea4 >2018-05-29 10:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [54b27274] FINISH, SetVdsStatusVDSCommand, log id: 6767eea4 >2018-05-29 10:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [54b27274] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 75a37b5e >2018-05-29 10:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [54b27274] FINISH, SetHaMaintenanceModeVDSCommand, log id: 75a37b5e >2018-05-29 10:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:05:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [54b27274] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:05:01,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7f8db494] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 66895f80 >2018-05-29 10:05:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [7f8db494] FINISH, GetHardwareInfoVDSCommand, log id: 66895f80 >2018-05-29 10:05:01,479-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [12171c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:05:01,483-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3e31b0cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:05:01,489-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7ed4e602] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:05:01,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7ed4e602] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70c036f3 >2018-05-29 10:05:02,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e551734] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b4e1d0e >2018-05-29 10:05:02,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e551734] FINISH, GetHardwareInfoVDSCommand, log id: 4b4e1d0e >2018-05-29 10:05:02,673-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5a885d2f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:05:02,678-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6f78221b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:05:02,683-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [363eb744] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:05:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [363eb744] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42011188 >2018-05-29 10:05:02,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:05:02,872-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7ed4e602] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70c036f3 >2018-05-29 10:05:02,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:02,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:02,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:02,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:02,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:02,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7ed4e602] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:02,888-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7ed4e602] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:05:02,891-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ed4e602] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ed4e602, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:05:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6707c711 >2018-05-29 10:05:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:05:02,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:05:02,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:05:02,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6707c711 >2018-05-29 10:05:02,998-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:05:03,011-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [58bcc1a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:05:03,012-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [58bcc1a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1cd9314d >2018-05-29 10:05:03,014-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [58bcc1a6] FINISH, SetVdsStatusVDSCommand, log id: 1cd9314d >2018-05-29 10:05:03,022-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58bcc1a6, Job ID: 09ee1422-77a9-41cb-a1f2-a9df21a84afc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:05:03,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3e31b0cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:05:03,029-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58bcc1a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3e31b0cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:05:03,039-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2b7c2245] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:05:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2b7c2245] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:05:03,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:05:03,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [363eb744] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42011188 >2018-05-29 10:05:03,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:03,080-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:03,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:03,085-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:03,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:05:03,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [363eb744] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:05:03,092-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [363eb744] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:05:03,095-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [363eb744] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 363eb744, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:05:03,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a09aa69 >2018-05-29 10:05:03,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:05:03,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:05:03,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:05:03,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a09aa69 >2018-05-29 10:05:03,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:05:03,212-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1c5fc324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:05:03,213-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1c5fc324] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ee42aca >2018-05-29 10:05:03,215-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1c5fc324] FINISH, SetVdsStatusVDSCommand, log id: 6ee42aca >2018-05-29 10:05:03,220-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c5fc324, Job ID: 42336737-a2aa-45cb-ab1f-9be459ccfe18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:05:03,225-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f78221b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:05:03,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1c5fc324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f78221b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:05:03,237-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2b4dd028] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:05:03,239-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2b4dd028] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [51b4b1da] Autorecovering 2 hosts >2018-05-29 10:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [51b4b1da] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18a16cff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b9a7e5d >2018-05-29 10:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18a16cff] FINISH, SetVdsStatusVDSCommand, log id: 3b9a7e5d >2018-05-29 10:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18a16cff] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18a16cff] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [67605445] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b40fa2 >2018-05-29 10:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [67605445] FINISH, SetVdsStatusVDSCommand, log id: 6b40fa2 >2018-05-29 10:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [67605445] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6cdbf39b >2018-05-29 10:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [67605445] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6cdbf39b >2018-05-29 10:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [67605445] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:10:01,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [de132eb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 524de3d >2018-05-29 10:10:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [de132eb] FINISH, GetHardwareInfoVDSCommand, log id: 524de3d >2018-05-29 10:10:01,091-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1c64338] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:10:01,099-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [70aefe3b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:10:01,104-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7427da44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:10:01,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7427da44] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab9964c >2018-05-29 10:10:01,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1079033a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5937d40c >2018-05-29 10:10:01,165-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1079033a] FINISH, GetHardwareInfoVDSCommand, log id: 5937d40c >2018-05-29 10:10:01,179-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [79ffb4bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:10:01,183-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5dc90903] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:10:01,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [58dde58f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:10:01,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [58dde58f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65df76d8 >2018-05-29 10:10:01,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:10:01,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7427da44] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab9964c >2018-05-29 10:10:01,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,500-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7427da44] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,512-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7427da44] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:10:01,515-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7427da44] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7427da44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:10:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7199db5a >2018-05-29 10:10:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:10:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:10:01,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [58dde58f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65df76d8 >2018-05-29 10:10:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,579-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:10:01,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [58dde58f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:10:01,591-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [58dde58f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:10:01,594-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [58dde58f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58dde58f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:10:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:10:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:10:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7199db5a >2018-05-29 10:10:01,620-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:10:01,622-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6234ce0c >2018-05-29 10:10:01,622-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:10:01,634-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6b5903c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:10:01,635-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b5903c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c12090 >2018-05-29 10:10:01,637-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b5903c1] FINISH, SetVdsStatusVDSCommand, log id: 41c12090 >2018-05-29 10:10:01,671-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5903c1, Job ID: b84fd694-47e1-4ffc-9e9f-b8813d6aa13e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:10:01,675-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70aefe3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:10:01,678-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6b5903c1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70aefe3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:10:01,689-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6f68a894] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:10:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6f68a894] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:10:01,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:10:01,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:10:01,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6234ce0c >2018-05-29 10:10:01,723-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:10:01,736-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1e4fe1f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:10:01,737-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e4fe1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 608ea763 >2018-05-29 10:10:01,739-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e4fe1f] FINISH, SetVdsStatusVDSCommand, log id: 608ea763 >2018-05-29 10:10:01,745-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e4fe1f, Job ID: 50a2927b-0fcf-4cda-b057-29e6cb3a1fc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:10:01,750-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dc90903, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:10:01,752-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e4fe1f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dc90903, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:10:01,762-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [44366838] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:10:01,763-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [44366838] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4406d76e] Autorecovering 2 hosts >2018-05-29 10:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4406d76e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:15:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4260da72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72a2a115 >2018-05-29 10:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4260da72] FINISH, SetVdsStatusVDSCommand, log id: 72a2a115 >2018-05-29 10:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4260da72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:15:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4260da72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2966256f >2018-05-29 10:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] FINISH, SetVdsStatusVDSCommand, log id: 2966256f >2018-05-29 10:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 530275b6 >2018-05-29 10:15:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4612a7ab] FINISH, SetHaMaintenanceModeVDSCommand, log id: 530275b6 >2018-05-29 10:15:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4612a7ab] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:15:02,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6350b44a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 73d50e5c >2018-05-29 10:15:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6350b44a] FINISH, GetHardwareInfoVDSCommand, log id: 73d50e5c >2018-05-29 10:15:02,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [41d0cf31] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44993ca6 >2018-05-29 10:15:02,514-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [41d0cf31] FINISH, GetHardwareInfoVDSCommand, log id: 44993ca6 >2018-05-29 10:15:02,521-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1b7ec227] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:15:02,527-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [14594626] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:15:02,530-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [68cfc8b4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:15:02,536-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [434c6540] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:15:02,536-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [17119ced] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:15:02,542-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4e17d5dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:15:02,549-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [434c6540] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 568c335 >2018-05-29 10:15:02,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4e17d5dd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 683e814b >2018-05-29 10:15:02,925-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:15:02,925-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4e17d5dd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 683e814b >2018-05-29 10:15:02,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:15:02,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [434c6540] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 568c335 >2018-05-29 10:15:02,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,929-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,930-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,931-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4e17d5dd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:15:02,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [434c6540] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:15:02,942-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4e17d5dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:15:02,944-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [434c6540] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:15:02,945-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4e17d5dd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e17d5dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:15:02,946-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [434c6540] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 434c6540, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:15:02,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2589e26e >2018-05-29 10:15:02,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:15:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:15:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:15:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:15:03,048-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:15:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2589e26e >2018-05-29 10:15:03,048-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:15:03,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50ba0d13 >2018-05-29 10:15:03,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:15:03,062-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [102d9ca0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:15:03,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [102d9ca0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17b4b4db >2018-05-29 10:15:03,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [102d9ca0] FINISH, SetVdsStatusVDSCommand, log id: 17b4b4db >2018-05-29 10:15:03,098-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 102d9ca0, Job ID: 9fcfb01c-43cb-40c8-a303-9f08f0934698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:15:03,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17119ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:15:03,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [102d9ca0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17119ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:15:03,116-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [292b573d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:15:03,117-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [292b573d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:15:03,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:15:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:15:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:15:03,151-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:15:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50ba0d13 >2018-05-29 10:15:03,152-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:15:03,164-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3f821d9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:15:03,165-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3f821d9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c132f1e >2018-05-29 10:15:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3f821d9b] FINISH, SetVdsStatusVDSCommand, log id: c132f1e >2018-05-29 10:15:03,173-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f821d9b, Job ID: b4d03afb-4e81-41f3-8f23-4adcb80dd391, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:15:03,177-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68cfc8b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:15:03,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3f821d9b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68cfc8b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:15:03,190-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3f5fdf35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:15:03,191-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3f5fdf35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [3b702f0d] Autorecovering 2 hosts >2018-05-29 10:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [3b702f0d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:20:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [56821dc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 675cc73e >2018-05-29 10:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [56821dc5] FINISH, SetVdsStatusVDSCommand, log id: 675cc73e >2018-05-29 10:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [56821dc5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:20:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [56821dc5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5a103525] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d32e639 >2018-05-29 10:20:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5a103525] FINISH, SetVdsStatusVDSCommand, log id: 1d32e639 >2018-05-29 10:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5a103525] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ca9abde >2018-05-29 10:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5a103525] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ca9abde >2018-05-29 10:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5a103525] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:20:00,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c037293] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42ecfa97 >2018-05-29 10:20:00,751-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [69343b5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 236bec59 >2018-05-29 10:20:00,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c037293] FINISH, GetHardwareInfoVDSCommand, log id: 42ecfa97 >2018-05-29 10:20:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [69343b5c] FINISH, GetHardwareInfoVDSCommand, log id: 236bec59 >2018-05-29 10:20:00,765-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6696dff4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:20:00,766-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [731413b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:20:00,770-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [170b3cb9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:20:00,775-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7243a8bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:20:00,775-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [54e09641] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:20:00,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5780f647] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:20:00,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7243a8bb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b2cccec >2018-05-29 10:20:00,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5780f647] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 784f5dd5 >2018-05-29 10:20:01,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:20:01,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7243a8bb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b2cccec >2018-05-29 10:20:01,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7243a8bb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,178-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7243a8bb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:20:01,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7243a8bb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7243a8bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:20:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 642060dd >2018-05-29 10:20:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:20:01,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:20:01,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5780f647] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 784f5dd5 >2018-05-29 10:20:01,220-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,221-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,225-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:20:01,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5780f647] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:20:01,233-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5780f647] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:20:01,236-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5780f647] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5780f647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:20:01,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:20:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:20:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:20:01,285-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:20:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 642060dd >2018-05-29 10:20:01,285-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:20:01,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11480024 >2018-05-29 10:20:01,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:20:01,298-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1208e14c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:20:01,299-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1208e14c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 113b79d7 >2018-05-29 10:20:01,301-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1208e14c] FINISH, SetVdsStatusVDSCommand, log id: 113b79d7 >2018-05-29 10:20:01,310-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1208e14c, Job ID: 6e33866c-9b36-4498-92cf-4e82f8703222, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:20:01,315-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 170b3cb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:20:01,317-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1208e14c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 170b3cb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:20:01,327-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [17609217] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:20:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [17609217] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:20:01,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:20:01,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:20:01,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11480024 >2018-05-29 10:20:01,382-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:20:01,394-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [58282f82] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:20:01,396-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58282f82] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f608ec4 >2018-05-29 10:20:01,397-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58282f82] FINISH, SetVdsStatusVDSCommand, log id: 4f608ec4 >2018-05-29 10:20:01,403-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58282f82, Job ID: 6426a1a7-fc6c-4d20-b935-e844e71093c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:20:01,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 54e09641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:20:01,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58282f82] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 54e09641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:20:01,420-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3290ba7a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:20:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3290ba7a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6472d5e6] Autorecovering 2 hosts >2018-05-29 10:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6472d5e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:25:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:25:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:25:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [d96f7ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ed9c962 >2018-05-29 10:25:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [d96f7ec] FINISH, SetVdsStatusVDSCommand, log id: ed9c962 >2018-05-29 10:25:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [d96f7ec] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:25:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [d96f7ec] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:25:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 180d91e1 >2018-05-29 10:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] FINISH, SetVdsStatusVDSCommand, log id: 180d91e1 >2018-05-29 10:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 37b6dca >2018-05-29 10:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2638ff3d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 37b6dca >2018-05-29 10:25:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2638ff3d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:25:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [24304e97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 248c0dfc >2018-05-29 10:25:02,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [24304e97] FINISH, GetHardwareInfoVDSCommand, log id: 248c0dfc >2018-05-29 10:25:02,105-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [532349c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:25:02,113-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [4602ba3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:25:02,120-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [304533f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:25:02,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [304533f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70dfe97d >2018-05-29 10:25:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [43eb1040] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 274726b >2018-05-29 10:25:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [43eb1040] FINISH, GetHardwareInfoVDSCommand, log id: 274726b >2018-05-29 10:25:02,148-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4014cf2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:25:02,153-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [413caaae] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:25:02,158-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [51ef3fb2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:25:02,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [51ef3fb2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f8d8d2 >2018-05-29 10:25:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:25:02,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [304533f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70dfe97d >2018-05-29 10:25:02,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:02,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:02,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:02,518-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:02,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:02,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [304533f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:02,525-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [304533f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:25:02,528-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [304533f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 304533f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:25:02,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75c01cef >2018-05-29 10:25:02,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:25:02,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:25:02,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:25:02,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75c01cef >2018-05-29 10:25:02,635-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:25:02,648-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [51a9c447] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:25:02,649-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51a9c447] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f2c6494 >2018-05-29 10:25:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [51a9c447] FINISH, SetVdsStatusVDSCommand, log id: 4f2c6494 >2018-05-29 10:25:02,685-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 51a9c447, Job ID: e0b635ba-638b-4327-ae56-a13a051678b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:25:02,690-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4602ba3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:25:02,692-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [51a9c447] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4602ba3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:25:02,705-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [387707f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:25:02,707-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [387707f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:25:03,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:25:03,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [51ef3fb2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f8d8d2 >2018-05-29 10:25:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:03,542-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:03,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:03,547-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:03,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:25:03,552-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [51ef3fb2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:25:03,556-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [51ef3fb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:25:03,559-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [51ef3fb2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51ef3fb2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:25:03,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17302ecd >2018-05-29 10:25:03,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:25:03,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:25:03,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:25:03,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17302ecd >2018-05-29 10:25:03,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:25:03,673-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [69d87ee1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:25:03,674-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [69d87ee1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6528a2ce >2018-05-29 10:25:03,677-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [69d87ee1] FINISH, SetVdsStatusVDSCommand, log id: 6528a2ce >2018-05-29 10:25:03,682-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69d87ee1, Job ID: e18cbe09-280f-4588-b790-97f065504cd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:25:03,687-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 413caaae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:25:03,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d87ee1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 413caaae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:25:03,699-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3857f01d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:25:03,701-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3857f01d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d07043f] Autorecovering 2 hosts >2018-05-29 10:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d07043f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:30:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ee8f4dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3836b300 >2018-05-29 10:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ee8f4dd] FINISH, SetVdsStatusVDSCommand, log id: 3836b300 >2018-05-29 10:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2ee8f4dd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:30:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ee8f4dd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55f66a56 >2018-05-29 10:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] FINISH, SetVdsStatusVDSCommand, log id: 55f66a56 >2018-05-29 10:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7ca3fbfe >2018-05-29 10:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [7ed4ba38] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7ca3fbfe >2018-05-29 10:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7ed4ba38] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:30:00,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cfa4b20] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 593d230f >2018-05-29 10:30:00,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cfa4b20] FINISH, GetHardwareInfoVDSCommand, log id: 593d230f >2018-05-29 10:30:00,288-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [380e1b13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:30:00,296-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1d9798db] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:30:00,302-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2078d137] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:30:00,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2078d137] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7648b9fd >2018-05-29 10:30:00,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:30:00,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2078d137] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7648b9fd >2018-05-29 10:30:00,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:00,704-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:00,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:00,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:00,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:00,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2078d137] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:00,716-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2078d137] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:30:00,719-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2078d137] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2078d137, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:30:00,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4cae0e22 >2018-05-29 10:30:00,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:30:00,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:30:00,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:30:00,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4cae0e22 >2018-05-29 10:30:00,824-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:30:00,838-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6a1addd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:30:00,840-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a1addd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4381fbb2 >2018-05-29 10:30:00,842-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a1addd] FINISH, SetVdsStatusVDSCommand, log id: 4381fbb2 >2018-05-29 10:30:00,873-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a1addd, Job ID: 78921673-736d-42c5-a943-84c3d196e8d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:30:00,878-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d9798db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:30:00,880-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a1addd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d9798db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:30:00,891-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [50cf7693] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:30:00,892-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [50cf7693] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:30:01,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e51b8b9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 750b5739 >2018-05-29 10:30:01,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e51b8b9] FINISH, GetHardwareInfoVDSCommand, log id: 750b5739 >2018-05-29 10:30:01,477-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [790d6a9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:30:01,482-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1aebf501] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:30:01,487-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [485ad85e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:30:01,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [485ad85e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c260e26 >2018-05-29 10:30:01,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:30:01,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [485ad85e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c260e26 >2018-05-29 10:30:01,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:01,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:01,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:01,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:01,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:30:01,875-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [485ad85e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:30:01,878-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [485ad85e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:30:01,880-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [485ad85e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 485ad85e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:30:01,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 182174bd >2018-05-29 10:30:01,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:30:01,980-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:30:01,980-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:30:01,980-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 182174bd >2018-05-29 10:30:01,980-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:30:01,993-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6fec5a73] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:30:01,995-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6fec5a73] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f99b5a6 >2018-05-29 10:30:01,997-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6fec5a73] FINISH, SetVdsStatusVDSCommand, log id: f99b5a6 >2018-05-29 10:30:02,002-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fec5a73, Job ID: 0462cd05-70db-42d6-aee6-143c2b8b08f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:30:02,007-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1aebf501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:30:02,009-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6fec5a73] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1aebf501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:30:02,019-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [505af95b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:30:02,020-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [505af95b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3dd1a564] Autorecovering 2 hosts >2018-05-29 10:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3dd1a564] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:35:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [604a98e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e16023d >2018-05-29 10:35:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [604a98e3] FINISH, SetVdsStatusVDSCommand, log id: 6e16023d >2018-05-29 10:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [604a98e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:35:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [604a98e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:35:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6e915364] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 480abed1 >2018-05-29 10:35:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6e915364] FINISH, SetVdsStatusVDSCommand, log id: 480abed1 >2018-05-29 10:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6e915364] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 391498d5 >2018-05-29 10:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6e915364] FINISH, SetHaMaintenanceModeVDSCommand, log id: 391498d5 >2018-05-29 10:35:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:35:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6e915364] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:35:01,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [10cb05f8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 184f6863 >2018-05-29 10:35:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [10cb05f8] FINISH, GetHardwareInfoVDSCommand, log id: 184f6863 >2018-05-29 10:35:01,781-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [36a4de55] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:35:01,789-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6ac7cd8e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:35:01,795-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [918de9c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:35:01,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [918de9c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dfa72d9 >2018-05-29 10:35:02,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:35:02,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [918de9c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dfa72d9 >2018-05-29 10:35:02,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:02,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:02,209-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:02,209-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [918de9c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:02,213-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [918de9c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:35:02,215-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [918de9c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 918de9c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:35:02,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cd4b8bd >2018-05-29 10:35:02,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:35:02,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:35:02,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:35:02,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cd4b8bd >2018-05-29 10:35:02,321-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:35:02,335-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4878fc3b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:35:02,336-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4878fc3b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a17031f >2018-05-29 10:35:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4878fc3b] FINISH, SetVdsStatusVDSCommand, log id: 5a17031f >2018-05-29 10:35:02,372-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4878fc3b, Job ID: 73554457-71c8-42c9-90af-6eef72801f6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:35:02,376-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ac7cd8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:35:02,379-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4878fc3b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ac7cd8e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:35:02,390-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [a1ee432] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:35:02,391-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [a1ee432] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:35:02,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [57ff2501] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 43c49bb1 >2018-05-29 10:35:02,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [57ff2501] FINISH, GetHardwareInfoVDSCommand, log id: 43c49bb1 >2018-05-29 10:35:02,831-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3bea6ddb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:35:02,835-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [73a3ce2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:35:02,841-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c870efc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:35:02,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c870efc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab81c4d >2018-05-29 10:35:03,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:35:03,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c870efc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab81c4d >2018-05-29 10:35:03,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:03,218-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:03,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:03,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:35:03,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c870efc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:35:03,230-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c870efc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:35:03,233-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c870efc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c870efc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:35:03,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c65a65 >2018-05-29 10:35:03,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:35:03,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:35:03,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:35:03,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c65a65 >2018-05-29 10:35:03,333-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:35:03,346-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7457f5dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:35:03,347-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7457f5dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3594744d >2018-05-29 10:35:03,350-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7457f5dc] FINISH, SetVdsStatusVDSCommand, log id: 3594744d >2018-05-29 10:35:03,355-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7457f5dc, Job ID: 400778d9-fc76-44fb-94b4-9f6ea8bd8044, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:35:03,360-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73a3ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:35:03,362-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7457f5dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73a3ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:35:03,372-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59fa878a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:35:03,374-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59fa878a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [599b8ac2] Autorecovering 2 hosts >2018-05-29 10:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [599b8ac2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:40:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:40:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [556648dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 114aa52f >2018-05-29 10:40:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [556648dd] FINISH, SetVdsStatusVDSCommand, log id: 114aa52f >2018-05-29 10:40:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [556648dd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:40:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [556648dd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:40:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11d69afc >2018-05-29 10:40:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] FINISH, SetVdsStatusVDSCommand, log id: 11d69afc >2018-05-29 10:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 15269ec7 >2018-05-29 10:40:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [ba8ea82] FINISH, SetHaMaintenanceModeVDSCommand, log id: 15269ec7 >2018-05-29 10:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [ba8ea82] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:40:01,045-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1924df55] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2bd4295b >2018-05-29 10:40:01,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1924df55] FINISH, GetHardwareInfoVDSCommand, log id: 2bd4295b >2018-05-29 10:40:01,060-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [47dc7b9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:40:01,068-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [496c50ca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:40:01,074-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41f698ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:40:01,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41f698ad] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 206d80d >2018-05-29 10:40:01,445-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:40:01,445-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41f698ad] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 206d80d >2018-05-29 10:40:01,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:01,449-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:01,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:01,454-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:01,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:01,458-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [41f698ad] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:01,461-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41f698ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:40:01,463-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41f698ad] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41f698ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:40:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e672d90 >2018-05-29 10:40:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:40:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:40:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:40:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e672d90 >2018-05-29 10:40:01,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:40:01,575-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6675e563] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:40:01,576-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6675e563] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3af3907 >2018-05-29 10:40:01,579-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6675e563] FINISH, SetVdsStatusVDSCommand, log id: 3af3907 >2018-05-29 10:40:01,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6675e563, Job ID: 086eb36a-dd0b-44cd-bd50-b63b6f746065, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:40:01,612-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 496c50ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:40:01,614-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6675e563] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 496c50ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:40:01,625-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [12928d85] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:40:01,626-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [12928d85] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:40:03,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e0dffcc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 71d3f3d2 >2018-05-29 10:40:03,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [6e0dffcc] FINISH, GetHardwareInfoVDSCommand, log id: 71d3f3d2 >2018-05-29 10:40:03,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [13acaf69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:40:03,098-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [271ec1dd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:40:03,104-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2201c548] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:40:03,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2201c548] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14d13051 >2018-05-29 10:40:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:40:03,489-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2201c548] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14d13051 >2018-05-29 10:40:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:03,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:03,498-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:03,499-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:03,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:40:03,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2201c548] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:40:03,508-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2201c548] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:40:03,510-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2201c548] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2201c548, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:40:03,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77d6cf58 >2018-05-29 10:40:03,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:40:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:40:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:40:03,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77d6cf58 >2018-05-29 10:40:03,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:40:03,623-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5bd223a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:40:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5bd223a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d4159f9 >2018-05-29 10:40:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5bd223a8] FINISH, SetVdsStatusVDSCommand, log id: d4159f9 >2018-05-29 10:40:03,632-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bd223a8, Job ID: 38da2e48-83b5-40e7-94e3-47a217d25728, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:40:03,637-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 271ec1dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:40:03,639-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5bd223a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 271ec1dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:40:03,649-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3f7d3c86] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:40:03,650-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3f7d3c86] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:45:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13d95a29] Autorecovering 2 hosts >2018-05-29 10:45:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13d95a29] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:45:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:45:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:45:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d162ca0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fde0393 >2018-05-29 10:45:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d162ca0] FINISH, SetVdsStatusVDSCommand, log id: 7fde0393 >2018-05-29 10:45:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4d162ca0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:45:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4d162ca0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:45:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:45:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:45:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60c6021a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f07d6e >2018-05-29 10:45:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60c6021a] FINISH, SetVdsStatusVDSCommand, log id: 30f07d6e >2018-05-29 10:45:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60c6021a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 69b187c1 >2018-05-29 10:45:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60c6021a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 69b187c1 >2018-05-29 10:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60c6021a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:45:01,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5f4c97b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 32476638 >2018-05-29 10:45:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [5f4c97b8] FINISH, GetHardwareInfoVDSCommand, log id: 32476638 >2018-05-29 10:45:01,276-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3ffacdad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:45:01,280-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [66ab26d8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:45:01,286-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [ac9129b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:45:01,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [ac9129b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e9d973b >2018-05-29 10:45:01,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:45:01,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [ac9129b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e9d973b >2018-05-29 10:45:01,725-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:01,726-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:01,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:01,731-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:01,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:01,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [ac9129b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:01,738-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [ac9129b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:45:01,741-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [ac9129b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ac9129b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:45:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 253a5ed0 >2018-05-29 10:45:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:45:01,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:45:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:45:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:45:01,848-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:45:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 253a5ed0 >2018-05-29 10:45:01,848-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:45:01,862-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [16b223b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:45:01,863-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16b223b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e3fa775 >2018-05-29 10:45:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [16b223b] FINISH, SetVdsStatusVDSCommand, log id: 1e3fa775 >2018-05-29 10:45:01,873-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16b223b, Job ID: 61d4eece-ce81-4724-8a25-d445b2878d19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:45:01,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66ab26d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:45:01,880-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16b223b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66ab26d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:45:01,891-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4bc4fb42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:45:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4bc4fb42] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:45:02,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2566c801] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f4f8788 >2018-05-29 10:45:02,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2566c801] FINISH, GetHardwareInfoVDSCommand, log id: 5f4f8788 >2018-05-29 10:45:02,324-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [67cd41d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:45:02,329-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5ca41c45] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:45:02,334-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3c04d2ec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:45:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3c04d2ec] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10305958 >2018-05-29 10:45:04,656-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:45:04,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3c04d2ec] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10305958 >2018-05-29 10:45:04,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:04,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:04,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:04,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:04,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:45:04,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3c04d2ec] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:45:04,674-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3c04d2ec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:45:04,676-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c04d2ec] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c04d2ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:45:04,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ca579c >2018-05-29 10:45:04,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:45:04,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:45:04,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:45:04,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:45:04,776-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:45:04,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ca579c >2018-05-29 10:45:04,777-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:45:04,790-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [46381caf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:45:04,791-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46381caf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f6f0503 >2018-05-29 10:45:04,794-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46381caf] FINISH, SetVdsStatusVDSCommand, log id: 5f6f0503 >2018-05-29 10:45:04,825-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46381caf, Job ID: 360d0409-3de8-48c9-84e9-83bfce8af333, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:45:04,830-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ca41c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:45:04,832-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46381caf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ca41c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:45:04,843-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [306ff3e3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:45:04,844-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [306ff3e3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [17fefa0b] Autorecovering 2 hosts >2018-05-29 10:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [17fefa0b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [74a41539] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69ce828a >2018-05-29 10:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [74a41539] FINISH, SetVdsStatusVDSCommand, log id: 69ce828a >2018-05-29 10:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [74a41539] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:50:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [74a41539] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:50:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2870e919 >2018-05-29 10:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] FINISH, SetVdsStatusVDSCommand, log id: 2870e919 >2018-05-29 10:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6bec82a0 >2018-05-29 10:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [46f05fa7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6bec82a0 >2018-05-29 10:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:50:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [46f05fa7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:50:02,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [962f45d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6da673ae >2018-05-29 10:50:02,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [962f45d] FINISH, GetHardwareInfoVDSCommand, log id: 6da673ae >2018-05-29 10:50:02,538-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [5780c8c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:50:02,544-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [30bbc828] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3f1266b2 >2018-05-29 10:50:02,545-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [30bbc828] FINISH, GetHardwareInfoVDSCommand, log id: 3f1266b2 >2018-05-29 10:50:02,547-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [58d710eb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:50:02,553-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [77a9fe92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:50:02,560-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [465e1275] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:50:02,564-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [50cfbad1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:50:02,570-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [450d5cba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:50:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [450d5cba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 666d6b50 >2018-05-29 10:50:02,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [77a9fe92] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 615f7765 >2018-05-29 10:50:02,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:50:02,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [450d5cba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 666d6b50 >2018-05-29 10:50:02,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:02,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:02,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:50:02,989-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [77a9fe92] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 615f7765 >2018-05-29 10:50:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:02,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [450d5cba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:02,993-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:02,994-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:02,995-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [450d5cba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:02,998-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:02,998-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [450d5cba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 450d5cba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:50:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:50:03,002-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [77a9fe92] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:50:03,005-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [77a9fe92] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:50:03,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e98fbdc >2018-05-29 10:50:03,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:50:03,008-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77a9fe92] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77a9fe92, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:50:03,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:50:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:50:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:50:03,105-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:50:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e98fbdc >2018-05-29 10:50:03,105-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:50:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cd9da0b >2018-05-29 10:50:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:50:03,118-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2fb734d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:50:03,119-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fb734d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19e7ec0a >2018-05-29 10:50:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2fb734d8] FINISH, SetVdsStatusVDSCommand, log id: 19e7ec0a >2018-05-29 10:50:03,155-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fb734d8, Job ID: 57401dda-8e93-4830-b457-8caeab68c7b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:50:03,159-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50cfbad1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:50:03,161-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fb734d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50cfbad1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:50:03,172-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [92d63b0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:50:03,173-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [92d63b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:50:03,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:50:03,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:50:03,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cd9da0b >2018-05-29 10:50:03,212-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:50:03,225-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [674043ec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:50:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [674043ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f8078a2 >2018-05-29 10:50:03,229-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [674043ec] FINISH, SetVdsStatusVDSCommand, log id: 1f8078a2 >2018-05-29 10:50:03,234-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 674043ec, Job ID: 3f362d54-5601-46d6-a203-444b59967aed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:50:03,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 58d710eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:50:03,241-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [674043ec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 58d710eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:50:03,251-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [144da548] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:50:03,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [144da548] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:54:42,991-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [336fb7b0] Attempting to update VMs/Templates Ovf. >2018-05-29 10:54:42,994-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 10:54:42,994-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 10:54:42,994-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:54:42,999-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 10:54:43,002-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 10:54:43,002-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 10:54:43,002-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 10:54:43,002-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 10:54:43,003-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 10:54:43,003-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [4fb39f49] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 10:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3be407c0] Autorecovering 2 hosts >2018-05-29 10:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3be407c0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:55:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:55:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:55:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49e36883] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7bb70d5d >2018-05-29 10:55:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49e36883] FINISH, SetVdsStatusVDSCommand, log id: 7bb70d5d >2018-05-29 10:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49e36883] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 10:55:00,068-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [49e36883] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:55:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:55:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 10:55:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,082-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56e740ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fd894ea >2018-05-29 10:55:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56e740ce] FINISH, SetVdsStatusVDSCommand, log id: 6fd894ea >2018-05-29 10:55:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [56e740ce] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 644aa55a >2018-05-29 10:55:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [56e740ce] FINISH, SetHaMaintenanceModeVDSCommand, log id: 644aa55a >2018-05-29 10:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 10:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56e740ce] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 10:55:00,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [21d28557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c33b7b1 >2018-05-29 10:55:00,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [21d28557] FINISH, GetHardwareInfoVDSCommand, log id: 5c33b7b1 >2018-05-29 10:55:00,841-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [29040ba7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f32e66c >2018-05-29 10:55:00,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [29040ba7] FINISH, GetHardwareInfoVDSCommand, log id: 5f32e66c >2018-05-29 10:55:00,846-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [42c87c56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:55:00,855-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [a8b467a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:55:00,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [46977de7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:55:00,861-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5ffc0509] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:55:00,865-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [584950e5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:55:00,870-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5b9e570] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 10:55:00,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5ffc0509] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64368a8 >2018-05-29 10:55:00,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5b9e570] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39df2bc6 >2018-05-29 10:55:01,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:55:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5b9e570] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39df2bc6 >2018-05-29 10:55:01,246-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,247-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 10:55:01,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5ffc0509] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64368a8 >2018-05-29 10:55:01,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,256-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,260-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5b9e570] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,263-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5b9e570] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:55:01,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 10:55:01,266-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5ffc0509] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 10:55:01,266-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b9e570] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b9e570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:55:01,269-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5ffc0509] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 10:55:01,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73813524 >2018-05-29 10:55:01,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:55:01,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5ffc0509] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ffc0509, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:55:01,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:55:01,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:55:01,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73813524 >2018-05-29 10:55:01,370-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:55:01,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9eb20d4 >2018-05-29 10:55:01,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 10:55:01,383-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [43eb8575] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:55:01,384-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43eb8575] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 490a9983 >2018-05-29 10:55:01,386-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43eb8575] FINISH, SetVdsStatusVDSCommand, log id: 490a9983 >2018-05-29 10:55:01,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43eb8575, Job ID: fc267b7e-a343-4ab3-b474-07e9b8540480, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:55:01,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 584950e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:55:01,424-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [43eb8575] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 584950e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:55:01,434-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [347879c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 10:55:01,435-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [347879c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 10:55:01,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:55:01,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 10:55:01,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 10:55:01,477-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 10:55:01,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9eb20d4 >2018-05-29 10:55:01,477-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 10:55:01,489-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2201dff8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:55:01,491-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2201dff8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42246adb >2018-05-29 10:55:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2201dff8] FINISH, SetVdsStatusVDSCommand, log id: 42246adb >2018-05-29 10:55:01,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2201dff8, Job ID: 43dc1931-9d2e-4a32-b6b5-59f0a6ce6469, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 10:55:01,502-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: a8b467a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 10:55:01,504-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2201dff8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: a8b467a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 10:55:01,515-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [4d639180] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 10:55:01,516-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4d639180] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:00:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c8f1ba6] Autorecovering 2 hosts >2018-05-29 11:00:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4c8f1ba6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:00:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6685af5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ce68d51 >2018-05-29 11:00:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6685af5c] FINISH, SetVdsStatusVDSCommand, log id: ce68d51 >2018-05-29 11:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6685af5c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:00:00,048-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6685af5c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:00:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:00:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:00:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60042523] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23007c79 >2018-05-29 11:00:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [60042523] FINISH, SetVdsStatusVDSCommand, log id: 23007c79 >2018-05-29 11:00:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60042523] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 77c143a6 >2018-05-29 11:00:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [60042523] FINISH, SetHaMaintenanceModeVDSCommand, log id: 77c143a6 >2018-05-29 11:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:00:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [60042523] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:00:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bbed291] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 172de13f >2018-05-29 11:00:02,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [38701ba9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 589a03df >2018-05-29 11:00:02,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bbed291] FINISH, GetHardwareInfoVDSCommand, log id: 172de13f >2018-05-29 11:00:02,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [38701ba9] FINISH, GetHardwareInfoVDSCommand, log id: 589a03df >2018-05-29 11:00:02,238-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [3e3129e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:00:02,239-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [52cd103a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:00:02,245-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5737561] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:00:02,248-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [65e4329] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:00:02,250-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7552f955] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:00:02,254-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3d431ade] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:00:02,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7552f955] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21e71fc8 >2018-05-29 11:00:02,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3d431ade] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3391cee0 >2018-05-29 11:00:02,644-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:00:02,644-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7552f955] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21e71fc8 >2018-05-29 11:00:02,648-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,649-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,653-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7552f955] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7552f955] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:00:02,663-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7552f955] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7552f955, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:00:02,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: e6b5b45 >2018-05-29 11:00:02,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:00:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:00:02,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3d431ade] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3391cee0 >2018-05-29 11:00:02,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:00:02,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3d431ade] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:00:02,696-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3d431ade] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:00:02,699-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3d431ade] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d431ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:00:02,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:00:02,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:00:02,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: e6b5b45 >2018-05-29 11:00:02,771-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:00:02,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c5137ec >2018-05-29 11:00:02,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:00:02,785-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7ddf80ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:00:02,786-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7ddf80ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 700f9e6d >2018-05-29 11:00:02,788-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7ddf80ca] FINISH, SetVdsStatusVDSCommand, log id: 700f9e6d >2018-05-29 11:00:02,796-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ddf80ca, Job ID: 2d4e9cdc-80a0-4b89-9fea-c1ed1d3ee315, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:00:02,801-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5737561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:00:02,804-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ddf80ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5737561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:00:02,814-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1699c707] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:00:02,815-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1699c707] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:00:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:00:02,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:00:02,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:00:02,870-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:00:02,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: c5137ec >2018-05-29 11:00:02,870-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:00:02,883-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [22fd26c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:00:02,884-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22fd26c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48ea90ad >2018-05-29 11:00:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22fd26c2] FINISH, SetVdsStatusVDSCommand, log id: 48ea90ad >2018-05-29 11:00:02,892-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22fd26c2, Job ID: 93d3c489-4dfa-4b65-b21a-3e06e9051fe2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:00:02,896-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 65e4329, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:00:02,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [22fd26c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 65e4329, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:00:02,908-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [9cc2aa4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:00:02,909-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [9cc2aa4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:05:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d2d1677] Autorecovering 2 hosts >2018-05-29 11:05:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d2d1677] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [21c110b2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7b126345 >2018-05-29 11:05:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [21c110b2] FINISH, SetVdsStatusVDSCommand, log id: 7b126345 >2018-05-29 11:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [21c110b2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:05:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [21c110b2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:05:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:05:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [79499da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e582cfa >2018-05-29 11:05:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [79499da] FINISH, SetVdsStatusVDSCommand, log id: 1e582cfa >2018-05-29 11:05:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [79499da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6888db54 >2018-05-29 11:05:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [79499da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6888db54 >2018-05-29 11:05:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:05:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [79499da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:05:00,445-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16db8535] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 14592e8e >2018-05-29 11:05:00,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16db8535] FINISH, GetHardwareInfoVDSCommand, log id: 14592e8e >2018-05-29 11:05:00,460-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7c98ad19] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:05:00,468-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [43d22ff7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:05:00,473-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [65ad710d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:05:00,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [65ad710d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c84cba0 >2018-05-29 11:05:00,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [a74b77f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7c19c363 >2018-05-29 11:05:00,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [a74b77f] FINISH, GetHardwareInfoVDSCommand, log id: 7c19c363 >2018-05-29 11:05:00,509-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [60e82dea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:05:00,513-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3d9ed7a1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:05:00,519-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dad4f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:05:00,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dad4f3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 542ff4c8 >2018-05-29 11:05:00,863-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:05:00,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [65ad710d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c84cba0 >2018-05-29 11:05:00,867-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,868-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [65ad710d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,879-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [65ad710d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:05:00,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [65ad710d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65ad710d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:05:00,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 765af215 >2018-05-29 11:05:00,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:05:00,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:05:00,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51dad4f3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 542ff4c8 >2018-05-29 11:05:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,904-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:05:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51dad4f3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:05:00,915-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51dad4f3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:05:00,918-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51dad4f3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51dad4f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:05:00,991-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:05:00,991-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:05:00,991-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 765af215 >2018-05-29 11:05:00,991-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:05:00,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 467a547c >2018-05-29 11:05:00,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:05:01,004-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2034a7ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:05:01,005-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2034a7ba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f98ad3f >2018-05-29 11:05:01,008-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2034a7ba] FINISH, SetVdsStatusVDSCommand, log id: 1f98ad3f >2018-05-29 11:05:01,040-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2034a7ba, Job ID: ae7a8a4a-f55c-4152-9025-9913a03eb596, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:05:01,045-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43d22ff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:05:01,047-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2034a7ba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43d22ff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:05:01,058-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [385609c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:05:01,059-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [385609c1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:05:01,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:05:01,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:05:01,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 467a547c >2018-05-29 11:05:01,096-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:05:01,109-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [336cd419] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:05:01,110-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [336cd419] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 129fa88d >2018-05-29 11:05:01,112-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [336cd419] FINISH, SetVdsStatusVDSCommand, log id: 129fa88d >2018-05-29 11:05:01,117-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 336cd419, Job ID: b7e7dded-68a4-4382-a5ff-646127b22848, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:05:01,122-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d9ed7a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:05:01,124-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [336cd419] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d9ed7a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:05:01,134-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [12559737] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:05:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [12559737] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [45fc4b29] Autorecovering 2 hosts >2018-05-29 11:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [45fc4b29] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e6baaf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13d05da7 >2018-05-29 11:10:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [e6baaf] FINISH, SetVdsStatusVDSCommand, log id: 13d05da7 >2018-05-29 11:10:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [e6baaf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:10:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [e6baaf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:10:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:10:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [caef888] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: dfc1893 >2018-05-29 11:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [caef888] FINISH, SetVdsStatusVDSCommand, log id: dfc1893 >2018-05-29 11:10:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [caef888] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 41a208fd >2018-05-29 11:10:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [caef888] FINISH, SetHaMaintenanceModeVDSCommand, log id: 41a208fd >2018-05-29 11:10:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:10:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [caef888] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:10:01,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3be8d7a7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: fe7d14e >2018-05-29 11:10:01,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3be8d7a7] FINISH, GetHardwareInfoVDSCommand, log id: fe7d14e >2018-05-29 11:10:01,711-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2b0b1b0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:10:01,719-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [66b2e8d4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:10:01,725-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a4014a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:10:01,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a4014a8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 98f63c5 >2018-05-29 11:10:01,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1507b31d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: dd0bf12 >2018-05-29 11:10:01,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1507b31d] FINISH, GetHardwareInfoVDSCommand, log id: dd0bf12 >2018-05-29 11:10:01,775-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [d988a95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:10:01,783-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2de64aa3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:10:01,788-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6efe4fc8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:10:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6efe4fc8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f788aa1 >2018-05-29 11:10:02,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:10:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6efe4fc8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f788aa1 >2018-05-29 11:10:02,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:02,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:02,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:02,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:02,233-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6efe4fc8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:02,236-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6efe4fc8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:10:02,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6efe4fc8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6efe4fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:10:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1411959a >2018-05-29 11:10:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:10:02,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:10:02,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:10:02,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1411959a >2018-05-29 11:10:02,343-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:10:02,357-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1205f3a7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:10:02,358-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1205f3a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50ab1644 >2018-05-29 11:10:02,361-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1205f3a7] FINISH, SetVdsStatusVDSCommand, log id: 50ab1644 >2018-05-29 11:10:02,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1205f3a7, Job ID: c079ef67-a886-43cc-9ecb-9dd4c745db84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:10:02,398-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2de64aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:10:02,400-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1205f3a7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2de64aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:10:02,411-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [2fe2f488] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:10:02,412-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [2fe2f488] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:10:03,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:10:03,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a4014a8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 98f63c5 >2018-05-29 11:10:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:03,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:03,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:03,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:03,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:10:03,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a4014a8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:10:03,074-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a4014a8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:10:03,077-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a4014a8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a4014a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:10:03,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 353760d6 >2018-05-29 11:10:03,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:10:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:10:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:10:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 353760d6 >2018-05-29 11:10:03,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:10:03,190-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6801c788] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:10:03,192-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6801c788] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d1c2b0e >2018-05-29 11:10:03,194-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6801c788] FINISH, SetVdsStatusVDSCommand, log id: 4d1c2b0e >2018-05-29 11:10:03,199-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6801c788, Job ID: 2c28e3aa-2ae2-4903-9ad5-7f13bf6a3c03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:10:03,204-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66b2e8d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:10:03,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6801c788] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66b2e8d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:10:03,217-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1467b4ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:10:03,218-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1467b4ec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [56f2f85f] Autorecovering 2 hosts >2018-05-29 11:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [56f2f85f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [521db0f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76770731 >2018-05-29 11:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [521db0f2] FINISH, SetVdsStatusVDSCommand, log id: 76770731 >2018-05-29 11:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [521db0f2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:15:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [521db0f2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [273135e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4350aca6 >2018-05-29 11:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [273135e] FINISH, SetVdsStatusVDSCommand, log id: 4350aca6 >2018-05-29 11:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [273135e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 32b46672 >2018-05-29 11:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [273135e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 32b46672 >2018-05-29 11:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:15:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [273135e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:15:00,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3e14ed94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11117c82 >2018-05-29 11:15:00,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3e14ed94] FINISH, GetHardwareInfoVDSCommand, log id: 11117c82 >2018-05-29 11:15:00,802-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [54a0a9c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:15:00,811-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7e94d017] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:15:00,816-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [30eead02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:15:00,829-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [30eead02] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2a5d9569 >2018-05-29 11:15:01,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:15:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [30eead02] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2a5d9569 >2018-05-29 11:15:01,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:01,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:01,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:01,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:01,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [30eead02] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:01,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [30eead02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:15:01,234-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [30eead02] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30eead02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:15:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1971277c >2018-05-29 11:15:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:15:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:15:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:15:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1971277c >2018-05-29 11:15:01,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:15:01,352-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [482daa08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:15:01,354-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [482daa08] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c53acb9 >2018-05-29 11:15:01,356-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [482daa08] FINISH, SetVdsStatusVDSCommand, log id: c53acb9 >2018-05-29 11:15:01,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 482daa08, Job ID: 3d14bd48-df2f-4a6a-8114-0e7baee8f39b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:15:01,391-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e94d017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:15:01,394-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [482daa08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e94d017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:15:01,405-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [50ca1bdd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:15:01,406-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [50ca1bdd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:15:03,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ec3d7ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 645b57 >2018-05-29 11:15:03,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ec3d7ed] FINISH, GetHardwareInfoVDSCommand, log id: 645b57 >2018-05-29 11:15:03,122-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7922bdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:15:03,127-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [57c980b1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:15:03,132-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [46b3347f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:15:03,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [46b3347f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a1106ac >2018-05-29 11:15:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:15:03,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [46b3347f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a1106ac >2018-05-29 11:15:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:15:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [46b3347f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:15:03,523-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [46b3347f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:15:03,526-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46b3347f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 46b3347f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:15:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2e5205 >2018-05-29 11:15:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:15:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:15:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:15:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2e5205 >2018-05-29 11:15:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:15:03,639-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3b4c998d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:15:03,640-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b4c998d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3eca43b3 >2018-05-29 11:15:03,642-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b4c998d] FINISH, SetVdsStatusVDSCommand, log id: 3eca43b3 >2018-05-29 11:15:03,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b4c998d, Job ID: 99ac2271-7d46-4f4e-a01e-69a285d791a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:15:03,652-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 57c980b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:15:03,654-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b4c998d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 57c980b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:15:03,665-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2705ac49] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:15:03,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2705ac49] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4443b56d] Autorecovering 2 hosts >2018-05-29 11:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4443b56d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2d3d0e8b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40db560a >2018-05-29 11:20:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2d3d0e8b] FINISH, SetVdsStatusVDSCommand, log id: 40db560a >2018-05-29 11:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2d3d0e8b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:20:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2d3d0e8b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7904973c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1530d2ac >2018-05-29 11:20:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7904973c] FINISH, SetVdsStatusVDSCommand, log id: 1530d2ac >2018-05-29 11:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7904973c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56b3f4a9 >2018-05-29 11:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7904973c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56b3f4a9 >2018-05-29 11:20:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7904973c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:20:01,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [433dbc7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f4f5336 >2018-05-29 11:20:01,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [433dbc7] FINISH, GetHardwareInfoVDSCommand, log id: 6f4f5336 >2018-05-29 11:20:01,352-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [13bddd27] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:20:01,361-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5bb12a08] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:20:01,366-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [35537a1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:20:01,377-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [35537a1d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3d4369b4 >2018-05-29 11:20:02,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4b6b97c5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: d736fa5 >2018-05-29 11:20:02,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4b6b97c5] FINISH, GetHardwareInfoVDSCommand, log id: d736fa5 >2018-05-29 11:20:02,281-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [113a3a33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:20:02,286-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2c3018b4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:20:02,291-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1a4936e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:20:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1a4936e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22ed814e >2018-05-29 11:20:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:20:02,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1a4936e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22ed814e >2018-05-29 11:20:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1a4936e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1a4936e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:20:02,695-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1a4936e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a4936e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:20:02,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f8f0d62 >2018-05-29 11:20:02,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:20:02,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:20:02,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [35537a1d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3d4369b4 >2018-05-29 11:20:02,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,753-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:20:02,761-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [35537a1d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:20:02,764-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [35537a1d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:20:02,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [35537a1d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 35537a1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:20:02,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:20:02,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:20:02,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f8f0d62 >2018-05-29 11:20:02,802-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:20:02,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e91d23f >2018-05-29 11:20:02,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:20:02,815-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [699a388a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:20:02,816-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [699a388a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9c05066 >2018-05-29 11:20:02,818-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [699a388a] FINISH, SetVdsStatusVDSCommand, log id: 9c05066 >2018-05-29 11:20:02,850-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 699a388a, Job ID: 909c7dd6-2355-49bb-88de-5c8377602227, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:20:02,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c3018b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:20:02,857-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [699a388a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c3018b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:20:02,867-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [64637d10] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:20:02,868-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [64637d10] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:20:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:20:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:20:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e91d23f >2018-05-29 11:20:02,903-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:20:02,915-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [53d7f324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:20:02,916-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53d7f324] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1aeb76ff >2018-05-29 11:20:02,918-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53d7f324] FINISH, SetVdsStatusVDSCommand, log id: 1aeb76ff >2018-05-29 11:20:02,924-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 53d7f324, Job ID: 64637357-da7d-4802-a1c8-9dc68286e281, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:20:02,928-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bb12a08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:20:02,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [53d7f324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bb12a08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:20:02,940-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [77aec4aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:20:02,941-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [77aec4aa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:25:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55f1e2e] Autorecovering 2 hosts >2018-05-29 11:25:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55f1e2e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:25:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:25:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:25:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [176b8a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5cc68a82 >2018-05-29 11:25:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [176b8a3] FINISH, SetVdsStatusVDSCommand, log id: 5cc68a82 >2018-05-29 11:25:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [176b8a3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:25:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [176b8a3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:25:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:25:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:25:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,118-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,120-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a4109dc >2018-05-29 11:25:00,123-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] FINISH, SetVdsStatusVDSCommand, log id: a4109dc >2018-05-29 11:25:00,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1196fd83 >2018-05-29 11:25:00,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b3a9f6e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1196fd83 >2018-05-29 11:25:00,128-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:25:00,130-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b3a9f6e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:25:00,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72aa3282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ee134af >2018-05-29 11:25:00,532-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72aa3282] FINISH, GetHardwareInfoVDSCommand, log id: 3ee134af >2018-05-29 11:25:00,545-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4059ff2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:25:00,554-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [34c21a44] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:25:00,560-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6c5b4d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:25:00,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6c5b4d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5c7acad4 >2018-05-29 11:25:00,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [576a558d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3105e80b >2018-05-29 11:25:00,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [576a558d] FINISH, GetHardwareInfoVDSCommand, log id: 3105e80b >2018-05-29 11:25:00,604-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [d15ce10] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:25:00,608-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [786ed5d8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:25:00,614-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52ca7968] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:25:00,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52ca7968] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5056fda >2018-05-29 11:25:00,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:25:00,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6c5b4d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5c7acad4 >2018-05-29 11:25:00,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,954-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:00,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,959-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:00,962-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,963-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6c5b4d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:00,965-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6c5b4d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:25:00,968-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c5b4d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c5b4d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:25:00,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 95c53d >2018-05-29 11:25:00,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:25:00,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:25:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52ca7968] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5056fda >2018-05-29 11:25:00,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:00,994-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,995-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:00,999-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:25:00,999-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52ca7968] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:25:01,002-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52ca7968] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:25:01,004-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52ca7968] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ca7968, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:25:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:25:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:25:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 95c53d >2018-05-29 11:25:01,072-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:25:01,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7dd1ee49 >2018-05-29 11:25:01,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:25:01,084-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [60b008ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:25:01,085-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [60b008ff] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d482290 >2018-05-29 11:25:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [60b008ff] FINISH, SetVdsStatusVDSCommand, log id: 3d482290 >2018-05-29 11:25:01,095-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60b008ff, Job ID: 9c51a2e1-dfe2-4f96-b0a4-f15589823550, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:25:01,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34c21a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:25:01,101-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [60b008ff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34c21a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:25:01,112-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [539f83f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:25:01,113-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [539f83f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:25:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:25:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:25:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7dd1ee49 >2018-05-29 11:25:01,172-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:25:01,184-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5cc5e848] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:25:01,185-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5cc5e848] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1130838f >2018-05-29 11:25:01,187-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5cc5e848] FINISH, SetVdsStatusVDSCommand, log id: 1130838f >2018-05-29 11:25:01,192-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5cc5e848, Job ID: cf187848-4438-4a21-afdb-7ddc681c2ff3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:25:01,197-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 786ed5d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:25:01,199-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5cc5e848] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 786ed5d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:25:01,209-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5d404089] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:25:01,210-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5d404089] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7694baf8] Autorecovering 2 hosts >2018-05-29 11:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7694baf8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:30:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:30:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:30:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [29e81195] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fec497c >2018-05-29 11:30:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [29e81195] FINISH, SetVdsStatusVDSCommand, log id: fec497c >2018-05-29 11:30:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [29e81195] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:30:00,053-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [29e81195] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:30:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67462207 >2018-05-29 11:30:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] FINISH, SetVdsStatusVDSCommand, log id: 67462207 >2018-05-29 11:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: ce6e6e7 >2018-05-29 11:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4a21ace0] FINISH, SetHaMaintenanceModeVDSCommand, log id: ce6e6e7 >2018-05-29 11:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:30:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4a21ace0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:30:01,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [209f2bbe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 28f4b98a >2018-05-29 11:30:01,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [209f2bbe] FINISH, GetHardwareInfoVDSCommand, log id: 28f4b98a >2018-05-29 11:30:01,782-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6dacfab2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:30:01,790-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [55a49ad3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:30:01,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [544d0d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ddeb1b4 >2018-05-29 11:30:01,796-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [543c9566] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:30:01,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [544d0d3] FINISH, GetHardwareInfoVDSCommand, log id: 1ddeb1b4 >2018-05-29 11:30:01,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [543c9566] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f6b1c9d >2018-05-29 11:30:01,810-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [28dfa796] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:30:01,818-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [ca8a8cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:30:01,823-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [36dd7a9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:30:01,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [36dd7a9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2bd231e5 >2018-05-29 11:30:02,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:30:02,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [543c9566] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f6b1c9d >2018-05-29 11:30:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:02,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:02,198-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:02,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:02,202-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [543c9566] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:02,205-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [543c9566] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:30:02,208-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [543c9566] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 543c9566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:30:02,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fb8c36 >2018-05-29 11:30:02,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:30:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:30:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:30:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fb8c36 >2018-05-29 11:30:02,314-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:30:02,350-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [39c5813f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:30:02,352-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [39c5813f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 509b215c >2018-05-29 11:30:02,354-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [39c5813f] FINISH, SetVdsStatusVDSCommand, log id: 509b215c >2018-05-29 11:30:02,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39c5813f, Job ID: e8528859-299c-4d68-b06d-22b9e63f2625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:30:02,390-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55a49ad3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:30:02,392-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [39c5813f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55a49ad3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:30:02,403-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [796663e5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:30:02,404-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [796663e5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:30:04,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:30:04,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [36dd7a9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2bd231e5 >2018-05-29 11:30:04,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:04,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:04,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:04,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:04,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:30:04,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [36dd7a9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:30:04,169-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [36dd7a9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:30:04,171-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36dd7a9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36dd7a9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:30:04,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1c3bd2dc >2018-05-29 11:30:04,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:30:04,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:30:04,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:30:04,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:30:04,272-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:30:04,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1c3bd2dc >2018-05-29 11:30:04,273-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:30:04,285-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4c0913c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:30:04,286-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4c0913c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64df96c8 >2018-05-29 11:30:04,288-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4c0913c0] FINISH, SetVdsStatusVDSCommand, log id: 64df96c8 >2018-05-29 11:30:04,293-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c0913c0, Job ID: ffc904fa-e69a-42fe-97ab-a44d3b94a445, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:30:04,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ca8a8cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:30:04,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c0913c0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ca8a8cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:30:04,310-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4f03beaf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:30:04,311-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4f03beaf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [559f9970] Autorecovering 2 hosts >2018-05-29 11:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [559f9970] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:35:00,011-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,018-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2a10c147] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f1efb8b >2018-05-29 11:35:00,020-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2a10c147] FINISH, SetVdsStatusVDSCommand, log id: 7f1efb8b >2018-05-29 11:35:00,022-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,023-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2a10c147] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:35:00,023-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2a10c147] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:35:00,028-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:35:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:35:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [44885b88] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65d98975 >2018-05-29 11:35:00,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [44885b88] FINISH, SetVdsStatusVDSCommand, log id: 65d98975 >2018-05-29 11:35:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [44885b88] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 74e96910 >2018-05-29 11:35:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [44885b88] FINISH, SetHaMaintenanceModeVDSCommand, log id: 74e96910 >2018-05-29 11:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:35:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [44885b88] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:35:02,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6f09751b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 698f0649 >2018-05-29 11:35:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6f09751b] FINISH, GetHardwareInfoVDSCommand, log id: 698f0649 >2018-05-29 11:35:02,083-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2aa13116] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:35:02,088-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6e719975] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:35:02,093-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6632127f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:35:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6632127f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5550525c >2018-05-29 11:35:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:35:02,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6632127f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5550525c >2018-05-29 11:35:02,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:02,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:02,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:02,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:02,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:02,488-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6632127f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:02,492-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6632127f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:35:02,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6632127f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6632127f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:35:02,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63fe5e34 >2018-05-29 11:35:02,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:35:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:35:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:35:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63fe5e34 >2018-05-29 11:35:02,600-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:35:02,613-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [17e9528f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:35:02,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17e9528f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54f4f4f0 >2018-05-29 11:35:02,617-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17e9528f] FINISH, SetVdsStatusVDSCommand, log id: 54f4f4f0 >2018-05-29 11:35:02,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 17e9528f, Job ID: 910292cf-aa77-43cd-a1c0-96144d35a942, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:35:02,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6e719975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:35:02,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17e9528f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6e719975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:35:02,665-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [25ee9ddf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:35:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [25ee9ddf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:35:02,991-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [46f5c1bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 21eec82e >2018-05-29 11:35:02,992-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [46f5c1bb] FINISH, GetHardwareInfoVDSCommand, log id: 21eec82e >2018-05-29 11:35:03,005-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [337f2d8b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:35:03,010-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [aac82c6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:35:03,015-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [66978c31] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:35:03,022-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [66978c31] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11906d51 >2018-05-29 11:35:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:35:03,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [66978c31] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11906d51 >2018-05-29 11:35:03,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:03,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:35:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [66978c31] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:35:03,419-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [66978c31] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:35:03,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [66978c31] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 66978c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:35:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 46758fce >2018-05-29 11:35:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:35:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 46758fce >2018-05-29 11:35:03,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:35:03,536-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [f7453fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:35:03,537-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7453fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30cc1e3b >2018-05-29 11:35:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7453fd] FINISH, SetVdsStatusVDSCommand, log id: 30cc1e3b >2018-05-29 11:35:03,545-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f7453fd, Job ID: 3b3fec15-7aa2-4467-b981-d09ed8b5ea07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:35:03,550-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: aac82c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:35:03,552-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f7453fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: aac82c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:35:03,563-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2c75b37f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:35:03,564-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2c75b37f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [75ea257d] Autorecovering 2 hosts >2018-05-29 11:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [75ea257d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1263edf6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1646240b >2018-05-29 11:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1263edf6] FINISH, SetVdsStatusVDSCommand, log id: 1646240b >2018-05-29 11:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1263edf6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1263edf6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 634cf1b4 >2018-05-29 11:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] FINISH, SetVdsStatusVDSCommand, log id: 634cf1b4 >2018-05-29 11:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 59b3da2b >2018-05-29 11:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [25e3e1f4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 59b3da2b >2018-05-29 11:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:40:00,102-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25e3e1f4] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 25e3e1f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 11:40:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [25e3e1f4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:40:00,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6385a0ec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 473f0bd8 >2018-05-29 11:40:00,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6385a0ec] FINISH, GetHardwareInfoVDSCommand, log id: 473f0bd8 >2018-05-29 11:40:00,428-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e209bc5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:40:00,436-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [304d9383] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:40:00,441-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [49769d36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:40:00,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [49769d36] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3709e760 >2018-05-29 11:40:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:40:00,833-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [49769d36] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3709e760 >2018-05-29 11:40:00,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:00,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:00,841-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:00,842-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:00,846-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [49769d36] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:00,849-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [49769d36] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:40:00,851-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49769d36] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49769d36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:40:00,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22253af2 >2018-05-29 11:40:00,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:40:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:40:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:40:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22253af2 >2018-05-29 11:40:00,956-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:40:00,969-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6b1efa68] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:40:00,971-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6b1efa68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14629bdb >2018-05-29 11:40:00,973-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6b1efa68] FINISH, SetVdsStatusVDSCommand, log id: 14629bdb >2018-05-29 11:40:01,004-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b1efa68, Job ID: 7fe9d545-645c-49a0-a27d-6b8e3e14068f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:40:01,008-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 304d9383, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:40:01,010-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6b1efa68] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 304d9383, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:40:01,021-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1a60b205] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:40:01,022-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1a60b205] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:40:01,191-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fc850d7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 571a9e37 >2018-05-29 11:40:01,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fc850d7] FINISH, GetHardwareInfoVDSCommand, log id: 571a9e37 >2018-05-29 11:40:01,205-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [633f8ad9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:40:01,209-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2aeb27ab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:40:01,214-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [16e73925] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:40:01,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [16e73925] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3d496fae >2018-05-29 11:40:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:40:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [16e73925] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3d496fae >2018-05-29 11:40:01,606-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:01,607-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:01,611-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:01,612-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:01,616-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:40:01,616-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [16e73925] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:40:01,620-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [16e73925] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:40:01,622-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16e73925] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 16e73925, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:40:01,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 160e2825 >2018-05-29 11:40:01,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:40:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:40:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:40:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 160e2825 >2018-05-29 11:40:01,720-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:40:01,733-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5e53f38c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:40:01,734-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e53f38c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b39ae52 >2018-05-29 11:40:01,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e53f38c] FINISH, SetVdsStatusVDSCommand, log id: 5b39ae52 >2018-05-29 11:40:01,742-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e53f38c, Job ID: 4d67eabb-a1d2-475a-a1ef-c23dd79940ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:40:01,747-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2aeb27ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:40:01,749-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e53f38c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2aeb27ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:40:01,759-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b0a708f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:40:01,760-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b0a708f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1f2a9a43] Autorecovering 2 hosts >2018-05-29 11:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1f2a9a43] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:45:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [c86477d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3601803b >2018-05-29 11:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [c86477d] FINISH, SetVdsStatusVDSCommand, log id: 3601803b >2018-05-29 11:45:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [c86477d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:45:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c86477d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:45:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d03eccc >2018-05-29 11:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] FINISH, SetVdsStatusVDSCommand, log id: 2d03eccc >2018-05-29 11:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 42870835 >2018-05-29 11:45:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [4dde0f6a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 42870835 >2018-05-29 11:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:45:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4dde0f6a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:45:01,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [47e5bbbc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e718fab >2018-05-29 11:45:01,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [47e5bbbc] FINISH, GetHardwareInfoVDSCommand, log id: 1e718fab >2018-05-29 11:45:01,686-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2ad096a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:45:01,694-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [167e6e43] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:45:01,699-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7060438d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:45:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7060438d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ca59fb >2018-05-29 11:45:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:45:02,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7060438d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ca59fb >2018-05-29 11:45:02,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,088-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7060438d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,100-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7060438d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:45:02,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7060438d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7060438d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:45:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c5e7c9 >2018-05-29 11:45:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:45:02,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:45:02,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:45:02,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c5e7c9 >2018-05-29 11:45:02,210-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:45:02,224-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [79cdd8fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:45:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79cdd8fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14975fb2 >2018-05-29 11:45:02,227-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79cdd8fd] FINISH, SetVdsStatusVDSCommand, log id: 14975fb2 >2018-05-29 11:45:02,258-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 79cdd8fd, Job ID: 0734a900-59f0-4c4c-8197-75d468a06275, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:45:02,263-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 167e6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:45:02,265-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79cdd8fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 167e6e43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:45:02,276-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [709469ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:45:02,277-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [709469ba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:45:02,377-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e49ddd5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dec8dae >2018-05-29 11:45:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e49ddd5] FINISH, GetHardwareInfoVDSCommand, log id: 3dec8dae >2018-05-29 11:45:02,390-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5587734d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:45:02,394-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5e1282b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:45:02,399-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3bfa770f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:45:02,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3bfa770f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 191870b4 >2018-05-29 11:45:02,784-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:45:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3bfa770f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 191870b4 >2018-05-29 11:45:02,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:45:02,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3bfa770f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:45:02,800-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3bfa770f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:45:02,803-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3bfa770f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bfa770f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:45:02,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f87541e >2018-05-29 11:45:02,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:45:02,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:45:02,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:45:02,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f87541e >2018-05-29 11:45:02,905-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:45:02,917-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [398d0fe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:45:02,918-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [398d0fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fc92018 >2018-05-29 11:45:02,920-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [398d0fe] FINISH, SetVdsStatusVDSCommand, log id: 7fc92018 >2018-05-29 11:45:02,926-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 398d0fe, Job ID: be29038a-7fda-4443-b4fe-09e53df8dc32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:45:02,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e1282b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:45:02,932-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [398d0fe] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e1282b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:45:02,942-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [dc7bb76] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:45:02,944-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [dc7bb76] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [fbffba2] Autorecovering 2 hosts >2018-05-29 11:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [fbffba2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d921e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e579e2d >2018-05-29 11:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d921e8] FINISH, SetVdsStatusVDSCommand, log id: 3e579e2d >2018-05-29 11:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d921e8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:50:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d921e8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:50:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1925461e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3aeddc4d >2018-05-29 11:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1925461e] FINISH, SetVdsStatusVDSCommand, log id: 3aeddc4d >2018-05-29 11:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1925461e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 45ae9698 >2018-05-29 11:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1925461e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 45ae9698 >2018-05-29 11:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:50:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1925461e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:50:00,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [13983791] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7f225951 >2018-05-29 11:50:00,570-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [13983791] FINISH, GetHardwareInfoVDSCommand, log id: 7f225951 >2018-05-29 11:50:00,584-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [29324ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:50:00,592-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [9b574cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:50:00,598-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [62e72956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:50:00,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [62e72956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e9b271 >2018-05-29 11:50:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:50:01,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [62e72956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e9b271 >2018-05-29 11:50:01,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:01,054-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:01,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:01,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [62e72956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:01,066-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [62e72956] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:50:01,069-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [62e72956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62e72956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:50:01,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45b41aa8 >2018-05-29 11:50:01,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:50:01,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:50:01,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:50:01,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45b41aa8 >2018-05-29 11:50:01,176-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:50:01,191-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3b01ada9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:50:01,193-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b01ada9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 230caca2 >2018-05-29 11:50:01,195-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3b01ada9] FINISH, SetVdsStatusVDSCommand, log id: 230caca2 >2018-05-29 11:50:01,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b01ada9, Job ID: facfbfd8-e355-4ae3-9694-7513e110bb03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:50:01,232-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 9b574cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:50:01,234-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3b01ada9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 9b574cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:50:01,245-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5086b37] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:50:01,246-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5086b37] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:50:02,928-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [496347f6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4f272bf3 >2018-05-29 11:50:02,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [496347f6] FINISH, GetHardwareInfoVDSCommand, log id: 4f272bf3 >2018-05-29 11:50:02,944-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [55ebb8af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:50:02,948-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6558f0d6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:50:02,953-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [28daa5a6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:50:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [28daa5a6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 709a28de >2018-05-29 11:50:05,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:50:05,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [28daa5a6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 709a28de >2018-05-29 11:50:05,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:05,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:05,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:05,281-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:05,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:50:05,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [28daa5a6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:50:05,290-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [28daa5a6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:50:05,293-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [28daa5a6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 28daa5a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:50:05,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2162651e >2018-05-29 11:50:05,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:50:05,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:50:05,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:50:05,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2162651e >2018-05-29 11:50:05,395-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:50:05,409-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6327a986] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:50:05,410-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6327a986] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 325e303d >2018-05-29 11:50:05,412-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6327a986] FINISH, SetVdsStatusVDSCommand, log id: 325e303d >2018-05-29 11:50:05,418-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6327a986, Job ID: 98ae14ab-1bae-45ad-9566-d8618b0bc2b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:50:05,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6558f0d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:50:05,424-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6327a986] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6558f0d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:50:05,435-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [51fc95da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:50:05,436-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [51fc95da] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:54:43,003-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [785a95ba] Attempting to update VMs/Templates Ovf. >2018-05-29 11:54:43,006-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 11:54:43,006-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 11:54:43,006-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:54:43,010-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 11:54:43,014-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 11:54:43,014-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 11:54:43,014-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 11:54:43,014-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 11:54:43,015-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 11:54:43,015-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7bb722f5] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 11:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [758d8d59] Autorecovering 2 hosts >2018-05-29 11:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [758d8d59] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:55:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2af7b478] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 764e28be >2018-05-29 11:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2af7b478] FINISH, SetVdsStatusVDSCommand, log id: 764e28be >2018-05-29 11:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,079-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2af7b478] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 2af7b478, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 11:55:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2af7b478] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 11:55:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2af7b478] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 11:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5541058 >2018-05-29 11:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] FINISH, SetVdsStatusVDSCommand, log id: 5541058 >2018-05-29 11:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2550d3d >2018-05-29 11:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [13e3aaa8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2550d3d >2018-05-29 11:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 11:55:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [13e3aaa8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 11:55:01,897-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ecb2358] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 168a0263 >2018-05-29 11:55:01,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ecb2358] FINISH, GetHardwareInfoVDSCommand, log id: 168a0263 >2018-05-29 11:55:01,912-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [458ab0fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:55:01,921-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [545a6061] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:55:01,926-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [42bba5da] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:55:01,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [42bba5da] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25e0ecf9 >2018-05-29 11:55:02,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:55:02,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [42bba5da] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25e0ecf9 >2018-05-29 11:55:02,334-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:02,335-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:02,339-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:02,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:02,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [42bba5da] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:02,346-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [42bba5da] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:55:02,349-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42bba5da] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42bba5da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:55:02,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 47da0b41 >2018-05-29 11:55:02,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:55:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:55:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:55:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 47da0b41 >2018-05-29 11:55:02,454-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:55:02,467-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [458d5b57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:55:02,468-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [458d5b57] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f7d18ec >2018-05-29 11:55:02,471-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [458d5b57] FINISH, SetVdsStatusVDSCommand, log id: 7f7d18ec >2018-05-29 11:55:02,503-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 458d5b57, Job ID: 029a4d24-59f1-4b30-b58b-43b6eb1e01f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:55:02,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 545a6061, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:55:02,509-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [458d5b57] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 545a6061, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:55:02,520-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2fc0a9ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 11:55:02,521-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2fc0a9ec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 11:55:03,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586ab90a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 50b45453 >2018-05-29 11:55:03,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586ab90a] FINISH, GetHardwareInfoVDSCommand, log id: 50b45453 >2018-05-29 11:55:03,206-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2064105f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:55:03,210-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2b2f1691] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:55:03,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [280b2ffc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 11:55:03,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [280b2ffc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41cf4358 >2018-05-29 11:55:03,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 11:55:03,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [280b2ffc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41cf4358 >2018-05-29 11:55:03,595-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:03,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:03,599-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:03,600-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 11:55:03,605-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [280b2ffc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 11:55:03,607-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [280b2ffc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 11:55:03,610-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [280b2ffc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 280b2ffc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 11:55:03,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5561ba49 >2018-05-29 11:55:03,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:55:03,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 11:55:03,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 11:55:03,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5561ba49 >2018-05-29 11:55:03,710-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 11:55:03,723-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [623acf29] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:55:03,724-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [623acf29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f6a470d >2018-05-29 11:55:03,726-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [623acf29] FINISH, SetVdsStatusVDSCommand, log id: 7f6a470d >2018-05-29 11:55:03,731-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 623acf29, Job ID: c93a1298-4ac9-4b42-8d7b-84788763c6bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 11:55:03,736-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2b2f1691, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 11:55:03,738-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [623acf29] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2b2f1691, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 11:55:03,748-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f2acba6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 11:55:03,749-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f2acba6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ab77b05] Autorecovering 2 hosts >2018-05-29 12:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ab77b05] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:00:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2f0a8315] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39e93de5 >2018-05-29 12:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2f0a8315] FINISH, SetVdsStatusVDSCommand, log id: 39e93de5 >2018-05-29 12:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2f0a8315] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:00:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2f0a8315] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:00:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e670de7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 109b49ed >2018-05-29 12:00:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e670de7] FINISH, SetVdsStatusVDSCommand, log id: 109b49ed >2018-05-29 12:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3e670de7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 65076c55 >2018-05-29 12:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [3e670de7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 65076c55 >2018-05-29 12:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:00:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3e670de7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:00:01,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [683e8200] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60a53ff9 >2018-05-29 12:00:01,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [683e8200] FINISH, GetHardwareInfoVDSCommand, log id: 60a53ff9 >2018-05-29 12:00:01,442-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2ead6570] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:00:01,447-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4903338] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:00:01,452-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a3cb68d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:00:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a3cb68d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33d11a66 >2018-05-29 12:00:01,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:00:01,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a3cb68d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33d11a66 >2018-05-29 12:00:01,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:01,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:01,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:01,834-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:01,838-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:01,839-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a3cb68d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:01,842-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a3cb68d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:00:01,844-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a3cb68d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a3cb68d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:00:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5961861e >2018-05-29 12:00:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:00:01,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:00:01,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:00:01,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5961861e >2018-05-29 12:00:01,944-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:00:01,957-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [380fb947] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:00:01,958-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [380fb947] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fdf6436 >2018-05-29 12:00:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [380fb947] FINISH, SetVdsStatusVDSCommand, log id: 1fdf6436 >2018-05-29 12:00:01,989-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 380fb947, Job ID: f3a0434d-6f12-4df2-9742-38c323a003ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:00:01,993-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4903338, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:00:01,995-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [380fb947] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4903338, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:00:02,006-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2783d80c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:00:02,007-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2783d80c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:00:03,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6f40738a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6738c5d6 >2018-05-29 12:00:03,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6f40738a] FINISH, GetHardwareInfoVDSCommand, log id: 6738c5d6 >2018-05-29 12:00:03,978-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [44bcc85a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:00:03,982-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [617bf27c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:00:03,987-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1f134f35] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:00:03,994-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1f134f35] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f4a9821 >2018-05-29 12:00:04,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:00:04,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [1f134f35] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f4a9821 >2018-05-29 12:00:04,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:04,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:04,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:04,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:04,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:00:04,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [1f134f35] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:00:04,382-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [1f134f35] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:00:04,384-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1f134f35] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1f134f35, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:00:04,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c22325 >2018-05-29 12:00:04,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:00:04,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:00:04,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:00:04,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:00:04,483-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:00:04,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c22325 >2018-05-29 12:00:04,483-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:00:04,496-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [108d94f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:00:04,497-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [108d94f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ed5ebf9 >2018-05-29 12:00:04,499-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [108d94f5] FINISH, SetVdsStatusVDSCommand, log id: 1ed5ebf9 >2018-05-29 12:00:04,504-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 108d94f5, Job ID: de3dc367-257b-43c1-a9ce-122661981828, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:00:04,509-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 617bf27c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:00:04,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [108d94f5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 617bf27c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:00:04,522-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [112929ee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:00:04,523-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [112929ee] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:05:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [993dc5e] Autorecovering 2 hosts >2018-05-29 12:05:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [993dc5e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:05:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [737c6699] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26e6b47a >2018-05-29 12:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [737c6699] FINISH, SetVdsStatusVDSCommand, log id: 26e6b47a >2018-05-29 12:05:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [737c6699] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:05:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [737c6699] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:05:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [69ad19df] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 386fd29c >2018-05-29 12:05:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [69ad19df] FINISH, SetVdsStatusVDSCommand, log id: 386fd29c >2018-05-29 12:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [69ad19df] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: c44a95f >2018-05-29 12:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [69ad19df] FINISH, SetHaMaintenanceModeVDSCommand, log id: c44a95f >2018-05-29 12:05:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [69ad19df] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:05:02,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1bd32ad0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 65f0a6ff >2018-05-29 12:05:02,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1bd32ad0] FINISH, GetHardwareInfoVDSCommand, log id: 65f0a6ff >2018-05-29 12:05:02,135-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [53a4f93f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:05:02,140-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [102bcfb0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:05:02,145-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [43f48b43] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:05:02,157-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [43f48b43] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d23c33a >2018-05-29 12:05:02,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:05:02,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [43f48b43] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d23c33a >2018-05-29 12:05:02,546-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:02,547-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:02,551-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:02,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [43f48b43] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:02,559-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [43f48b43] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:05:02,561-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [43f48b43] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43f48b43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:05:02,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fde3c9 >2018-05-29 12:05:02,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:05:02,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [10338438] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1cce29ac >2018-05-29 12:05:02,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [10338438] FINISH, GetHardwareInfoVDSCommand, log id: 1cce29ac >2018-05-29 12:05:02,650-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4e2a1e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:05:02,655-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [637c6e0a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:05:02,660-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [71c81af9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:05:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [71c81af9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f0fb3d >2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:05:02,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:05:02,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:05:02,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fde3c9 >2018-05-29 12:05:02,670-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:05:02,682-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5acea721] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:05:02,683-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5acea721] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9354fa0 >2018-05-29 12:05:02,685-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5acea721] FINISH, SetVdsStatusVDSCommand, log id: 9354fa0 >2018-05-29 12:05:02,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5acea721, Job ID: d3819187-1d71-4c1f-8340-bb0c40db2e6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:05:02,698-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 102bcfb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:05:02,700-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5acea721] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 102bcfb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:05:02,711-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [783ca7bb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:05:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [783ca7bb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:05:03,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:05:03,033-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [71c81af9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f0fb3d >2018-05-29 12:05:03,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:03,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:03,041-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:03,042-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:03,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:05:03,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [71c81af9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:05:03,050-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [71c81af9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:05:03,053-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c81af9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 71c81af9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:05:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74f2ddd4 >2018-05-29 12:05:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:05:03,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:05:03,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:05:03,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74f2ddd4 >2018-05-29 12:05:03,153-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:05:03,165-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [229aa01b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:05:03,166-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [229aa01b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d8be7a2 >2018-05-29 12:05:03,168-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [229aa01b] FINISH, SetVdsStatusVDSCommand, log id: 1d8be7a2 >2018-05-29 12:05:03,173-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 229aa01b, Job ID: bef77851-5681-4ef0-a641-6ce9e8f87d2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:05:03,178-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 637c6e0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:05:03,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [229aa01b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 637c6e0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:05:03,190-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1260fde0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:05:03,191-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1260fde0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5ae296d2] Autorecovering 2 hosts >2018-05-29 12:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5ae296d2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:10:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [413e43a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c11b0d0 >2018-05-29 12:10:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [413e43a0] FINISH, SetVdsStatusVDSCommand, log id: c11b0d0 >2018-05-29 12:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [413e43a0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:10:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [413e43a0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:10:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2de024c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dd1807e >2018-05-29 12:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2de024c1] FINISH, SetVdsStatusVDSCommand, log id: 2dd1807e >2018-05-29 12:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2de024c1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 732ee0c5 >2018-05-29 12:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2de024c1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 732ee0c5 >2018-05-29 12:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:10:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2de024c1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:10:00,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2fd5aff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2013359c >2018-05-29 12:10:00,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2fd5aff] FINISH, GetHardwareInfoVDSCommand, log id: 2013359c >2018-05-29 12:10:00,500-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [c7d7abc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:10:00,508-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [30172e38] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:10:00,513-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [53eb145b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:10:00,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [53eb145b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 352cfd2e >2018-05-29 12:10:00,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [734d7d01] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5ed92562 >2018-05-29 12:10:00,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [734d7d01] FINISH, GetHardwareInfoVDSCommand, log id: 5ed92562 >2018-05-29 12:10:00,827-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [10e6300c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:10:00,832-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7d0ea3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:10:00,837-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [14225c0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:10:00,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [14225c0f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47784b5e >2018-05-29 12:10:00,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:10:00,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [53eb145b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 352cfd2e >2018-05-29 12:10:00,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:00,900-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:00,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:00,904-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:00,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:00,909-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [53eb145b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:00,911-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [53eb145b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:10:00,914-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [53eb145b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53eb145b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:10:00,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7732b4c3 >2018-05-29 12:10:00,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:10:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:10:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:10:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:10:01,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:10:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7732b4c3 >2018-05-29 12:10:01,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:10:01,027-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [5d6b417c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:10:01,028-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5d6b417c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b6c80f1 >2018-05-29 12:10:01,030-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5d6b417c] FINISH, SetVdsStatusVDSCommand, log id: 1b6c80f1 >2018-05-29 12:10:01,062-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d6b417c, Job ID: 28136b99-0c76-485d-8d8d-d204bbb3cc4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:10:01,067-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 30172e38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:10:01,069-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d6b417c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 30172e38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:10:01,080-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [488cdfef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:10:01,081-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [488cdfef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:10:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:10:01,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [14225c0f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47784b5e >2018-05-29 12:10:01,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:01,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:01,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:01,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:01,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:10:01,232-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [14225c0f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:10:01,235-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [14225c0f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:10:01,237-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [14225c0f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 14225c0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:10:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3aa4f029 >2018-05-29 12:10:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:10:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:10:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:10:01,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3aa4f029 >2018-05-29 12:10:01,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:10:01,352-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [221d2b70] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:10:01,353-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [221d2b70] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 56866361 >2018-05-29 12:10:01,355-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [221d2b70] FINISH, SetVdsStatusVDSCommand, log id: 56866361 >2018-05-29 12:10:01,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 221d2b70, Job ID: aba05037-398d-458d-8c36-65f9edf42f3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:10:01,365-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d0ea3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:10:01,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [221d2b70] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d0ea3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:10:01,378-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1735156c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:10:01,379-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1735156c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c889a30] Autorecovering 2 hosts >2018-05-29 12:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [c889a30] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [416eb396] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3237f29a >2018-05-29 12:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [416eb396] FINISH, SetVdsStatusVDSCommand, log id: 3237f29a >2018-05-29 12:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [416eb396] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:15:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [416eb396] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:15:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [618689fc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 508b4501 >2018-05-29 12:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [618689fc] FINISH, SetVdsStatusVDSCommand, log id: 508b4501 >2018-05-29 12:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [618689fc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71405537 >2018-05-29 12:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [618689fc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71405537 >2018-05-29 12:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [618689fc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:15:01,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [50f8348c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78a854a7 >2018-05-29 12:15:01,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [50f8348c] FINISH, GetHardwareInfoVDSCommand, log id: 78a854a7 >2018-05-29 12:15:01,763-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7a2360e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:15:01,768-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3b92d40e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:15:01,773-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [201a4ac9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:15:01,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [201a4ac9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24aad26 >2018-05-29 12:15:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b48ab50] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76c791b5 >2018-05-29 12:15:02,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b48ab50] FINISH, GetHardwareInfoVDSCommand, log id: 76c791b5 >2018-05-29 12:15:02,028-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2b14a755] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:15:02,033-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4de88ef5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:15:02,038-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b03b8b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:15:02,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b03b8b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a4aaa8c >2018-05-29 12:15:02,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:15:02,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [201a4ac9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24aad26 >2018-05-29 12:15:02,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [201a4ac9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,230-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [201a4ac9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:15:02,232-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [201a4ac9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 201a4ac9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:15:02,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1145060 >2018-05-29 12:15:02,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:15:02,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:15:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:15:02,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:15:02,338-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:15:02,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1145060 >2018-05-29 12:15:02,339-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:15:02,351-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6d5ed5c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:15:02,353-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d5ed5c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 137b1da >2018-05-29 12:15:02,355-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d5ed5c7] FINISH, SetVdsStatusVDSCommand, log id: 137b1da >2018-05-29 12:15:02,363-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6d5ed5c7, Job ID: bf7a8e96-1e4c-4c2f-bc46-d6f2e66a8d97, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:15:02,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b92d40e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:15:02,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6d5ed5c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b92d40e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:15:02,380-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [53ec8626] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:15:02,381-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [53ec8626] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:15:02,404-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:15:02,404-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b03b8b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a4aaa8c >2018-05-29 12:15:02,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,414-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:15:02,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b03b8b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:15:02,421-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b03b8b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:15:02,424-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b03b8b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b03b8b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:15:02,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4bae770 >2018-05-29 12:15:02,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:15:02,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:15:02,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:15:02,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4bae770 >2018-05-29 12:15:02,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:15:02,535-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [16f855e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:15:02,536-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [16f855e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 698f942b >2018-05-29 12:15:02,538-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [16f855e6] FINISH, SetVdsStatusVDSCommand, log id: 698f942b >2018-05-29 12:15:02,544-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16f855e6, Job ID: 455f60e1-8c4d-4bc3-8296-24baa60202fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:15:02,548-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4de88ef5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:15:02,550-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [16f855e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4de88ef5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:15:02,561-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [37b67e16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:15:02,562-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [37b67e16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18d97500] Autorecovering 2 hosts >2018-05-29 12:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18d97500] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:20:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1a3f5dda] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61b52d74 >2018-05-29 12:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1a3f5dda] FINISH, SetVdsStatusVDSCommand, log id: 61b52d74 >2018-05-29 12:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1a3f5dda] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:20:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1a3f5dda] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:20:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7534137c >2018-05-29 12:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] FINISH, SetVdsStatusVDSCommand, log id: 7534137c >2018-05-29 12:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6acb7327 >2018-05-29 12:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62b9b9d9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6acb7327 >2018-05-29 12:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:20:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62b9b9d9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:20:02,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6b88039] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62b2dd5 >2018-05-29 12:20:02,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6b88039] FINISH, GetHardwareInfoVDSCommand, log id: 62b2dd5 >2018-05-29 12:20:02,978-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [116f3666] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:20:02,987-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [9fa88bd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:20:02,992-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4419d877] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:20:03,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4419d877] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12e675fa >2018-05-29 12:20:03,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19106d56] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 46368f1d >2018-05-29 12:20:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19106d56] FINISH, GetHardwareInfoVDSCommand, log id: 46368f1d >2018-05-29 12:20:03,238-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [47bc2f6e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:20:03,243-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2c828e9e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:20:03,247-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [78e6caed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:20:03,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [78e6caed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: cafb29 >2018-05-29 12:20:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:20:03,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4419d877] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12e675fa >2018-05-29 12:20:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4419d877] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,404-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4419d877] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:20:03,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4419d877] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4419d877, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:20:03,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6369e426 >2018-05-29 12:20:03,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:20:03,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:20:03,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:20:03,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6369e426 >2018-05-29 12:20:03,512-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:20:03,525-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [20a47d02] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:20:03,526-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20a47d02] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 586e1679 >2018-05-29 12:20:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20a47d02] FINISH, SetVdsStatusVDSCommand, log id: 586e1679 >2018-05-29 12:20:03,559-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20a47d02, Job ID: 0f532906-5987-43a2-8f40-bc6f0aefdb6f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:20:03,564-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 9fa88bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:20:03,566-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [20a47d02] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 9fa88bd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:20:03,577-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [f38e08c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:20:03,578-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [f38e08c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:20:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:20:03,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [78e6caed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: cafb29 >2018-05-29 12:20:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:20:03,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [78e6caed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:20:03,641-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [78e6caed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:20:03,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [78e6caed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78e6caed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:20:03,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 377f5cfa >2018-05-29 12:20:03,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:20:03,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:20:03,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:20:03,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 377f5cfa >2018-05-29 12:20:03,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:20:03,758-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [5d8a2e90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:20:03,759-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d8a2e90] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36c91031 >2018-05-29 12:20:03,761-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d8a2e90] FINISH, SetVdsStatusVDSCommand, log id: 36c91031 >2018-05-29 12:20:03,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d8a2e90, Job ID: a16fb409-9cbe-4c0e-be56-5ffbc102fe61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:20:03,771-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c828e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:20:03,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5d8a2e90] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c828e9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:20:03,783-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7710e8f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:20:03,784-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7710e8f5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [24482dcb] Autorecovering 2 hosts >2018-05-29 12:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [24482dcb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:25:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22b9c274] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55ebc8d6 >2018-05-29 12:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22b9c274] FINISH, SetVdsStatusVDSCommand, log id: 55ebc8d6 >2018-05-29 12:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [22b9c274] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:25:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [22b9c274] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:25:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:25:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:25:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,139-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d552b80] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6211db5e >2018-05-29 12:25:00,141-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4d552b80] FINISH, SetVdsStatusVDSCommand, log id: 6211db5e >2018-05-29 12:25:00,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4d552b80] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 54077e08 >2018-05-29 12:25:00,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4d552b80] FINISH, SetHaMaintenanceModeVDSCommand, log id: 54077e08 >2018-05-29 12:25:00,146-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:25:00,148-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4d552b80] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:25:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [498430c7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3688719e >2018-05-29 12:25:01,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [498430c7] FINISH, GetHardwareInfoVDSCommand, log id: 3688719e >2018-05-29 12:25:01,192-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7e0237fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:25:01,200-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2702321f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:25:01,206-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [a7d2544] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:25:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [a7d2544] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ea2bd36 >2018-05-29 12:25:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7ebacfc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1905f9d0 >2018-05-29 12:25:01,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7ebacfc6] FINISH, GetHardwareInfoVDSCommand, log id: 1905f9d0 >2018-05-29 12:25:01,436-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2554633b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:25:01,440-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [eb34907] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:25:01,445-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [af1b419] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:25:01,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [af1b419] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20ef7937 >2018-05-29 12:25:01,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:25:01,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [a7d2544] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ea2bd36 >2018-05-29 12:25:01,598-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:01,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:01,603-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:01,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:01,608-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [a7d2544] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:01,610-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [a7d2544] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:25:01,613-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [a7d2544] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a7d2544, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:25:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 49084988 >2018-05-29 12:25:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:25:01,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:25:01,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:25:01,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 49084988 >2018-05-29 12:25:01,716-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:25:01,730-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [33caa6ec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:25:01,731-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [33caa6ec] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77830bae >2018-05-29 12:25:01,733-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [33caa6ec] FINISH, SetVdsStatusVDSCommand, log id: 77830bae >2018-05-29 12:25:01,765-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33caa6ec, Job ID: b903b8ed-bbf6-4dbd-9aea-ae07163e44d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:25:01,770-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2702321f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:25:01,772-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [33caa6ec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2702321f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:25:01,783-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [71c12d85] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:25:01,784-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [71c12d85] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:25:02,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:25:02,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [af1b419] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20ef7937 >2018-05-29 12:25:02,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:02,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:02,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:02,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:02,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:25:02,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [af1b419] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:25:02,835-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [af1b419] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:25:02,837-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [af1b419] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: af1b419, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:25:02,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d74c080 >2018-05-29 12:25:02,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:25:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:25:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:25:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:25:02,936-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:25:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d74c080 >2018-05-29 12:25:02,937-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:25:02,950-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5dec1320] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:25:02,951-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5dec1320] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a89c1a >2018-05-29 12:25:02,953-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5dec1320] FINISH, SetVdsStatusVDSCommand, log id: 17a89c1a >2018-05-29 12:25:02,984-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5dec1320, Job ID: 1e0aa27d-ce96-4652-91ab-16c1c22f2560, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:25:02,988-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: eb34907, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:25:02,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5dec1320] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: eb34907, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:25:03,001-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [b5274c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:25:03,002-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [b5274c9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [284865f] Autorecovering 2 hosts >2018-05-29 12:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [284865f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7f8212e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b7e38e1 >2018-05-29 12:30:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7f8212e6] FINISH, SetVdsStatusVDSCommand, log id: 3b7e38e1 >2018-05-29 12:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7f8212e6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:30:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [7f8212e6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [37c95b31] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc8d05c >2018-05-29 12:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [37c95b31] FINISH, SetVdsStatusVDSCommand, log id: 2fc8d05c >2018-05-29 12:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [37c95b31] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7fcb2b1a >2018-05-29 12:30:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [37c95b31] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7fcb2b1a >2018-05-29 12:30:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:30:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [37c95b31] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:30:00,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3d1d6f64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: f4d157e >2018-05-29 12:30:00,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3d1d6f64] FINISH, GetHardwareInfoVDSCommand, log id: f4d157e >2018-05-29 12:30:00,675-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [57d1258c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:30:00,684-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [73553c34] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:30:00,689-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [25b18251] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:30:00,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [25b18251] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3362dc97 >2018-05-29 12:30:01,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:30:01,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [25b18251] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3362dc97 >2018-05-29 12:30:01,083-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:01,084-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:01,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:01,088-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:01,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:01,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [25b18251] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:01,096-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [25b18251] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:30:01,098-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [25b18251] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 25b18251, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:30:01,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ffee3a1 >2018-05-29 12:30:01,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:30:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:30:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:30:01,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: ffee3a1 >2018-05-29 12:30:01,202-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:30:01,216-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1af099a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:30:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1af099a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10cb0179 >2018-05-29 12:30:01,219-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1af099a5] FINISH, SetVdsStatusVDSCommand, log id: 10cb0179 >2018-05-29 12:30:01,251-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1af099a5, Job ID: 5116faa6-078e-49d2-98f1-dfefee8720c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:30:01,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73553c34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:30:01,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1af099a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73553c34, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:30:01,268-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [283089cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:30:01,269-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [283089cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:30:02,364-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [67cc8a2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8ba1d92 >2018-05-29 12:30:02,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [67cc8a2d] FINISH, GetHardwareInfoVDSCommand, log id: 8ba1d92 >2018-05-29 12:30:02,378-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [714649c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:30:02,383-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [3712739a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:30:02,388-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69be7ac4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:30:02,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69be7ac4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7693d110 >2018-05-29 12:30:02,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:30:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69be7ac4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7693d110 >2018-05-29 12:30:02,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:02,780-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:02,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:02,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:02,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:30:02,789-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69be7ac4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:30:02,792-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69be7ac4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:30:02,795-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69be7ac4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69be7ac4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:30:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e8962e7 >2018-05-29 12:30:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:30:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:30:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:30:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e8962e7 >2018-05-29 12:30:02,898-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:30:02,911-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [347de3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:30:02,912-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [347de3f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17b17efc >2018-05-29 12:30:02,914-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [347de3f] FINISH, SetVdsStatusVDSCommand, log id: 17b17efc >2018-05-29 12:30:02,920-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 347de3f, Job ID: 0f86cc41-7561-42d3-aa98-a3e4a7b87f99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:30:02,924-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3712739a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:30:02,927-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [347de3f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3712739a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:30:02,937-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1761d0a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:30:02,938-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1761d0a6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:35:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2bfbdc3] Autorecovering 2 hosts >2018-05-29 12:35:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2bfbdc3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:35:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49774037] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50541ebb >2018-05-29 12:35:00,082-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49774037] FINISH, SetVdsStatusVDSCommand, log id: 50541ebb >2018-05-29 12:35:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [49774037] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:35:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [49774037] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:35:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5440a16c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5150af53 >2018-05-29 12:35:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5440a16c] FINISH, SetVdsStatusVDSCommand, log id: 5150af53 >2018-05-29 12:35:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5440a16c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1bbf1298 >2018-05-29 12:35:00,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5440a16c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1bbf1298 >2018-05-29 12:35:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:35:00,106-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5440a16c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:35:00,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [561faec1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24a1143d >2018-05-29 12:35:00,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [561faec1] FINISH, GetHardwareInfoVDSCommand, log id: 24a1143d >2018-05-29 12:35:00,566-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [56033040] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:35:00,574-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [47828da9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:35:00,579-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6956d1f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:35:00,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6956d1f3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 84a5f10 >2018-05-29 12:35:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:35:00,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [6956d1f3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 84a5f10 >2018-05-29 12:35:00,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:00,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:00,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:00,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [6956d1f3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:00,991-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [6956d1f3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:35:00,993-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6956d1f3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6956d1f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:35:01,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7793e879 >2018-05-29 12:35:01,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:35:01,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:35:01,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:35:01,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7793e879 >2018-05-29 12:35:01,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:35:01,113-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [43dcac6e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:35:01,115-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43dcac6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e16a1fb >2018-05-29 12:35:01,117-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43dcac6e] FINISH, SetVdsStatusVDSCommand, log id: 4e16a1fb >2018-05-29 12:35:01,149-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43dcac6e, Job ID: c138ca4b-e69c-4914-8e69-2a5c29c43493, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:35:01,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47828da9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:35:01,156-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43dcac6e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47828da9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:35:01,167-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6b072dd0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:35:01,168-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6b072dd0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:35:02,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [d17218f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: bec6f00 >2018-05-29 12:35:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [d17218f] FINISH, GetHardwareInfoVDSCommand, log id: bec6f00 >2018-05-29 12:35:02,018-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7b9c0dbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:35:02,026-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [483d03e8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:35:02,031-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c14dfd7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:35:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c14dfd7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 723975fb >2018-05-29 12:35:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:35:03,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c14dfd7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 723975fb >2018-05-29 12:35:03,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:03,419-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:03,423-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:03,424-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:03,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:35:03,428-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c14dfd7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:35:03,431-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c14dfd7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:35:03,434-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c14dfd7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c14dfd7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:35:03,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3962c88f >2018-05-29 12:35:03,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:35:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:35:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:35:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3962c88f >2018-05-29 12:35:03,531-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:35:03,545-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [735add89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:35:03,546-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [735add89] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19b9a236 >2018-05-29 12:35:03,548-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [735add89] FINISH, SetVdsStatusVDSCommand, log id: 19b9a236 >2018-05-29 12:35:03,554-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 735add89, Job ID: 063279ec-1118-4693-b7b2-13855c747244, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:35:03,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 483d03e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:35:03,561-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [735add89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 483d03e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:35:03,571-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2469b916] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:35:03,572-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2469b916] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [b264439] Autorecovering 2 hosts >2018-05-29 12:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [b264439] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:40:00,031-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17113a2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca9dc4a >2018-05-29 12:40:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17113a2c] FINISH, SetVdsStatusVDSCommand, log id: 6ca9dc4a >2018-05-29 12:40:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17113a2c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:40:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [17113a2c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:40:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:40:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:40:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [563147d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3eb894b5 >2018-05-29 12:40:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [563147d6] FINISH, SetVdsStatusVDSCommand, log id: 3eb894b5 >2018-05-29 12:40:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [563147d6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 70dc8e36 >2018-05-29 12:40:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [563147d6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 70dc8e36 >2018-05-29 12:40:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [563147d6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:40:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [32a9491e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3c0ee727 >2018-05-29 12:40:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [32a9491e] FINISH, GetHardwareInfoVDSCommand, log id: 3c0ee727 >2018-05-29 12:40:01,339-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3c5f916c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:40:01,347-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1b2d1d89] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:40:01,353-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6a76aa3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:40:01,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6a76aa3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4188d33e >2018-05-29 12:40:01,728-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:40:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6a76aa3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4188d33e >2018-05-29 12:40:01,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:01,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:01,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6a76aa3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:01,744-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6a76aa3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:40:01,747-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a76aa3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6a76aa3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:40:01,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45036e64 >2018-05-29 12:40:01,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:40:01,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19c0cad2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 67349e7d >2018-05-29 12:40:01,794-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [19c0cad2] FINISH, GetHardwareInfoVDSCommand, log id: 67349e7d >2018-05-29 12:40:01,806-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [56ba86d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:40:01,810-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3f829868] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:40:01,815-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [18770b6f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:40:01,821-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [18770b6f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20a1a1a8 >2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:40:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:40:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:40:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45036e64 >2018-05-29 12:40:01,848-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:40:01,862-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3354cef9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:40:01,863-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3354cef9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f9bf088 >2018-05-29 12:40:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3354cef9] FINISH, SetVdsStatusVDSCommand, log id: 5f9bf088 >2018-05-29 12:40:01,895-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3354cef9, Job ID: 0a588721-10cf-4c4c-82bc-3fc15596e981, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:40:01,899-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b2d1d89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:40:01,901-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3354cef9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b2d1d89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:40:01,912-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [14601506] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:40:01,913-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [14601506] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:40:02,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:40:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [18770b6f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20a1a1a8 >2018-05-29 12:40:02,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:02,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:02,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:02,207-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:02,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:40:02,212-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [18770b6f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:40:02,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [18770b6f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:40:02,217-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18770b6f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18770b6f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:40:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b0706b7 >2018-05-29 12:40:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:40:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:40:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:40:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b0706b7 >2018-05-29 12:40:02,315-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:40:02,329-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1b1ec9ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:40:02,330-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b1ec9ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3797622e >2018-05-29 12:40:02,332-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b1ec9ae] FINISH, SetVdsStatusVDSCommand, log id: 3797622e >2018-05-29 12:40:02,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b1ec9ae, Job ID: 4a47734c-f9ac-4b4b-be8a-e431f100364f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:40:02,342-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f829868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:40:02,344-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b1ec9ae] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f829868, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:40:02,355-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6cd638a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:40:02,356-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6cd638a7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [57e2197f] Autorecovering 2 hosts >2018-05-29 12:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [57e2197f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:45:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1a6eb640] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d8d59e >2018-05-29 12:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1a6eb640] FINISH, SetVdsStatusVDSCommand, log id: 2d8d59e >2018-05-29 12:45:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1a6eb640] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:45:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1a6eb640] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:45:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [558cf7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e5053c5 >2018-05-29 12:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [558cf7b] FINISH, SetVdsStatusVDSCommand, log id: 6e5053c5 >2018-05-29 12:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [558cf7b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1782dd9d >2018-05-29 12:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [558cf7b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1782dd9d >2018-05-29 12:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:45:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [558cf7b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:45:02,593-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [121c4e1c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3fe7c78c >2018-05-29 12:45:02,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [121c4e1c] FINISH, GetHardwareInfoVDSCommand, log id: 3fe7c78c >2018-05-29 12:45:02,608-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [777e7dd5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:45:02,616-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1d1ea1fa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:45:02,621-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [693315a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:45:02,634-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [693315a9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16ed3f2d >2018-05-29 12:45:02,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27606eab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3c09caa4 >2018-05-29 12:45:02,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27606eab] FINISH, GetHardwareInfoVDSCommand, log id: 3c09caa4 >2018-05-29 12:45:02,962-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [a6fab29] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:45:02,966-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6d486be6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:45:02,971-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ffd623f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:45:02,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ffd623f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 46bd5f6b >2018-05-29 12:45:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:45:02,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [693315a9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16ed3f2d >2018-05-29 12:45:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,007-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,012-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [693315a9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,014-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [693315a9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:45:03,017-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [693315a9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 693315a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:45:03,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cc3fb89 >2018-05-29 12:45:03,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:45:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:45:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:45:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cc3fb89 >2018-05-29 12:45:03,122-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:45:03,136-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [5d97cd2a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:45:03,137-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5d97cd2a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51088bef >2018-05-29 12:45:03,139-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5d97cd2a] FINISH, SetVdsStatusVDSCommand, log id: 51088bef >2018-05-29 12:45:03,172-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d97cd2a, Job ID: 2816a803-7a75-435f-9f0a-1f94a323ce75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:45:03,177-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d1ea1fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:45:03,179-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d97cd2a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d1ea1fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:45:03,189-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [778ace6e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:45:03,190-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [778ace6e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:45:03,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:45:03,350-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ffd623f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 46bd5f6b >2018-05-29 12:45:03,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,355-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:45:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ffd623f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:45:03,366-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ffd623f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:45:03,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ffd623f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ffd623f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:45:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1767b82c >2018-05-29 12:45:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:45:03,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:45:03,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:45:03,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1767b82c >2018-05-29 12:45:03,473-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:45:03,486-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [555ac4b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:45:03,487-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555ac4b9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d3b06b6 >2018-05-29 12:45:03,490-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [555ac4b9] FINISH, SetVdsStatusVDSCommand, log id: 7d3b06b6 >2018-05-29 12:45:03,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 555ac4b9, Job ID: 575ee1f8-9a96-47f2-bc56-ba55d0d40b81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:45:03,500-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d486be6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:45:03,502-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [555ac4b9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d486be6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:45:03,513-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [34740f34] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:45:03,514-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [34740f34] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1ba04a93] Autorecovering 2 hosts >2018-05-29 12:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1ba04a93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a2cd813] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5449997b >2018-05-29 12:50:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a2cd813] FINISH, SetVdsStatusVDSCommand, log id: 5449997b >2018-05-29 12:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a2cd813] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:50:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a2cd813] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:50:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46aa70b1 >2018-05-29 12:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] FINISH, SetVdsStatusVDSCommand, log id: 46aa70b1 >2018-05-29 12:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1212c07 >2018-05-29 12:50:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2f15c8da] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1212c07 >2018-05-29 12:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:50:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2f15c8da] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:50:00,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [59babb5d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 65fc67df >2018-05-29 12:50:00,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [59babb5d] FINISH, GetHardwareInfoVDSCommand, log id: 65fc67df >2018-05-29 12:50:00,921-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [245875cd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:50:00,930-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [189942a5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:50:00,935-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [192aebf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:50:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [192aebf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3db81714 >2018-05-29 12:50:01,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6b931838] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4c86ebf4 >2018-05-29 12:50:01,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6b931838] FINISH, GetHardwareInfoVDSCommand, log id: 4c86ebf4 >2018-05-29 12:50:01,117-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [57c57f31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:50:01,121-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [602d77b4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:50:01,126-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [234b7ade] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:50:01,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [234b7ade] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d0e106f >2018-05-29 12:50:01,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:50:01,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [192aebf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3db81714 >2018-05-29 12:50:01,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [192aebf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,343-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [192aebf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:50:01,346-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [192aebf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 192aebf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:50:01,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b14e3d >2018-05-29 12:50:01,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:50:01,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:50:01,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:50:01,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b14e3d >2018-05-29 12:50:01,450-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:50:01,463-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1e8d9a0b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:50:01,464-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e8d9a0b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46e68d54 >2018-05-29 12:50:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1e8d9a0b] FINISH, SetVdsStatusVDSCommand, log id: 46e68d54 >2018-05-29 12:50:01,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e8d9a0b, Job ID: 41c2339b-fc93-4d62-863f-d856008f6504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:50:01,503-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 189942a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:50:01,505-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1e8d9a0b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 189942a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:50:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:50:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [234b7ade] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7d0e106f >2018-05-29 12:50:01,515-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2bc57482] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:50:01,516-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2bc57482] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:50:01,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:50:01,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [234b7ade] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:50:01,529-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [234b7ade] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:50:01,532-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234b7ade] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 234b7ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:50:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5fbcbf5b >2018-05-29 12:50:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:50:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:50:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:50:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5fbcbf5b >2018-05-29 12:50:01,632-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:50:01,645-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3702e7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:50:01,646-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3702e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1922315d >2018-05-29 12:50:01,648-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3702e7] FINISH, SetVdsStatusVDSCommand, log id: 1922315d >2018-05-29 12:50:01,654-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3702e7, Job ID: d2eec3d7-2e3c-4ac6-b11a-7ef4f3715a83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:50:01,658-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 602d77b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:50:01,660-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3702e7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 602d77b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:50:01,671-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7b924a89] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:50:01,672-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7b924a89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:54:43,016-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [5519621c] Attempting to update VMs/Templates Ovf. >2018-05-29 12:54:43,019-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 12:54:43,019-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 12:54:43,020-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:54:43,024-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 12:54:43,028-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 12:54:43,028-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 12:54:43,028-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 12:54:43,028-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 12:54:43,029-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 12:54:43,029-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [78712499] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 12:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7a28deaf] Autorecovering 2 hosts >2018-05-29 12:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7a28deaf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:55:00,031-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:55:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:55:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1ba2bd76] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 585eb811 >2018-05-29 12:55:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1ba2bd76] FINISH, SetVdsStatusVDSCommand, log id: 585eb811 >2018-05-29 12:55:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1ba2bd76] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 12:55:00,044-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ba2bd76] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:55:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:55:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 12:55:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 635a06df >2018-05-29 12:55:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] FINISH, SetVdsStatusVDSCommand, log id: 635a06df >2018-05-29 12:55:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 372d807c >2018-05-29 12:55:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [f6f4dcc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 372d807c >2018-05-29 12:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 12:55:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [f6f4dcc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 12:55:02,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6b6c5124] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 633eb109 >2018-05-29 12:55:02,156-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6b6c5124] FINISH, GetHardwareInfoVDSCommand, log id: 633eb109 >2018-05-29 12:55:02,169-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3f4349ab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:55:02,177-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [789b7002] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:55:02,182-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4b5a11e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:55:02,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4b5a11e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 523aaa1c >2018-05-29 12:55:02,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [24e5a684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6b11071d >2018-05-29 12:55:02,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [24e5a684] FINISH, GetHardwareInfoVDSCommand, log id: 6b11071d >2018-05-29 12:55:02,269-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [68f4b6aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:55:02,274-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1ae4d5ae] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:55:02,279-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [419e0371] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 12:55:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [419e0371] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1dad0c4f >2018-05-29 12:55:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:55:02,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [419e0371] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1dad0c4f >2018-05-29 12:55:02,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:02,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:02,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:02,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:02,684-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [419e0371] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:02,687-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [419e0371] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:55:02,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e0371] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 419e0371, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:55:02,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22373935 >2018-05-29 12:55:02,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:55:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:55:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:55:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22373935 >2018-05-29 12:55:02,797-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:55:02,810-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [251d481e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:55:02,811-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [251d481e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 804e2cd >2018-05-29 12:55:02,814-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [251d481e] FINISH, SetVdsStatusVDSCommand, log id: 804e2cd >2018-05-29 12:55:02,848-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 251d481e, Job ID: 42f9ca34-59cf-4bed-b754-51652223c5b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:55:02,853-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ae4d5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:55:02,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [251d481e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ae4d5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:55:02,866-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [699965d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 12:55:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [699965d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 12:55:03,559-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 12:55:03,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4b5a11e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 523aaa1c >2018-05-29 12:55:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:03,563-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:03,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 12:55:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4b5a11e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 12:55:03,575-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4b5a11e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 12:55:03,578-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4b5a11e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4b5a11e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 12:55:03,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a67fb81 >2018-05-29 12:55:03,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:55:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 12:55:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 12:55:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a67fb81 >2018-05-29 12:55:03,680-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 12:55:03,692-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4911c361] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:55:03,694-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4911c361] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43d32ff0 >2018-05-29 12:55:03,696-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4911c361] FINISH, SetVdsStatusVDSCommand, log id: 43d32ff0 >2018-05-29 12:55:03,701-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4911c361, Job ID: f1c376a7-273f-4ff4-9f64-e11770d26782, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 12:55:03,706-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 789b7002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 12:55:03,708-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4911c361] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 789b7002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 12:55:03,718-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [70c0cff8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 12:55:03,719-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [70c0cff8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6e6ef36d] Autorecovering 2 hosts >2018-05-29 13:00:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6e6ef36d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [110a0f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e911d2 >2018-05-29 13:00:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [110a0f3] FINISH, SetVdsStatusVDSCommand, log id: 5e911d2 >2018-05-29 13:00:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [110a0f3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:00:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [110a0f3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:00:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1d91a023] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 178a3c67 >2018-05-29 13:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1d91a023] FINISH, SetVdsStatusVDSCommand, log id: 178a3c67 >2018-05-29 13:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1d91a023] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 190a2184 >2018-05-29 13:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1d91a023] FINISH, SetHaMaintenanceModeVDSCommand, log id: 190a2184 >2018-05-29 13:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:00:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1d91a023] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:00:00,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6068bdce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7ddb1d60 >2018-05-29 13:00:00,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [6068bdce] FINISH, GetHardwareInfoVDSCommand, log id: 7ddb1d60 >2018-05-29 13:00:00,505-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [750f8f0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:00:00,513-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4544d1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:00:00,518-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [316001c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:00:00,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [316001c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17866d2d >2018-05-29 13:00:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:00:00,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [316001c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17866d2d >2018-05-29 13:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:00,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:00,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:00,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:00,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:00,927-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [316001c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:00,930-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [316001c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:00:00,934-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [316001c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 316001c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:00:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1dfc1e1b >2018-05-29 13:00:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:00:01,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:00:01,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:00:01,037-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1dfc1e1b >2018-05-29 13:00:01,037-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:00:01,051-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6259e36a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:00:01,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6259e36a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7006711b >2018-05-29 13:00:01,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6259e36a] FINISH, SetVdsStatusVDSCommand, log id: 7006711b >2018-05-29 13:00:01,086-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6259e36a, Job ID: 65c37659-df71-4c10-b64e-782dcc2cd805, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:00:01,091-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4544d1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:00:01,093-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6259e36a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4544d1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:00:01,103-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [69b80439] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:00:01,104-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [69b80439] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:00:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3662aae4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 300967c8 >2018-05-29 13:00:01,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3662aae4] FINISH, GetHardwareInfoVDSCommand, log id: 300967c8 >2018-05-29 13:00:01,396-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [275dfcc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:00:01,401-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1ea93fc8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:00:01,406-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3059539e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:00:01,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3059539e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7238a330 >2018-05-29 13:00:01,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:00:01,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3059539e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7238a330 >2018-05-29 13:00:01,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:01,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:01,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:01,798-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:01,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:00:01,802-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3059539e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:00:01,804-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3059539e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:00:01,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3059539e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3059539e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:00:01,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 8d343e7 >2018-05-29 13:00:01,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:00:01,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:00:01,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:00:01,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 8d343e7 >2018-05-29 13:00:01,916-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:00:01,929-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4b720646] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:00:01,930-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4b720646] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f008477 >2018-05-29 13:00:01,932-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4b720646] FINISH, SetVdsStatusVDSCommand, log id: 4f008477 >2018-05-29 13:00:01,965-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b720646, Job ID: 0da18e1c-4424-4e41-b390-66497acf0513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:00:01,970-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ea93fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:00:01,972-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4b720646] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ea93fc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:00:01,982-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3ada6569] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:00:01,983-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3ada6569] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5fd41efb] Autorecovering 2 hosts >2018-05-29 13:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5fd41efb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b299fc9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 727ef815 >2018-05-29 13:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b299fc9] FINISH, SetVdsStatusVDSCommand, log id: 727ef815 >2018-05-29 13:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b299fc9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7b299fc9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b2b119] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70bbed18 >2018-05-29 13:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [54b2b119] FINISH, SetVdsStatusVDSCommand, log id: 70bbed18 >2018-05-29 13:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b2b119] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23df1aed >2018-05-29 13:05:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [54b2b119] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23df1aed >2018-05-29 13:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:05:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [54b2b119] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:05:01,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [403d5d1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2aa2ac18 >2018-05-29 13:05:01,694-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [403d5d1] FINISH, GetHardwareInfoVDSCommand, log id: 2aa2ac18 >2018-05-29 13:05:01,707-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [325d4d02] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:05:01,715-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [17f700d2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:05:01,720-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7219cca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:05:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7219cca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 667e6507 >2018-05-29 13:05:02,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:05:02,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7219cca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 667e6507 >2018-05-29 13:05:02,122-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:02,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:02,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:02,127-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:02,131-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:02,132-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7219cca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:02,134-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7219cca] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:05:02,137-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7219cca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7219cca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:05:02,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c28e27f >2018-05-29 13:05:02,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:05:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:05:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:05:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c28e27f >2018-05-29 13:05:02,244-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:05:02,258-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [45bf7a1e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:05:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45bf7a1e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4381e6a0 >2018-05-29 13:05:02,261-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45bf7a1e] FINISH, SetVdsStatusVDSCommand, log id: 4381e6a0 >2018-05-29 13:05:02,295-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45bf7a1e, Job ID: 7a34a253-4105-4ffb-8e71-189c7db7c110, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:05:02,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17f700d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:05:02,302-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [45bf7a1e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17f700d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:05:02,313-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [e8bea6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:05:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [e8bea6a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:05:02,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10ae557e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 14f521fd >2018-05-29 13:05:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10ae557e] FINISH, GetHardwareInfoVDSCommand, log id: 14f521fd >2018-05-29 13:05:02,724-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4042d581] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:05:02,729-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [64f725c1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:05:02,734-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5fcbfea0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:05:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5fcbfea0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 449b1053 >2018-05-29 13:05:05,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:05:05,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5fcbfea0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 449b1053 >2018-05-29 13:05:05,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:05,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:05,070-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:05,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:05,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:05:05,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5fcbfea0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:05:05,080-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5fcbfea0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:05:05,082-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5fcbfea0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5fcbfea0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:05:05,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25fbe065 >2018-05-29 13:05:05,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:05:05,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:05:05,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:05:05,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25fbe065 >2018-05-29 13:05:05,184-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:05:05,197-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [38032122] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:05:05,198-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38032122] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ea8f0f >2018-05-29 13:05:05,200-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [38032122] FINISH, SetVdsStatusVDSCommand, log id: 31ea8f0f >2018-05-29 13:05:05,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38032122, Job ID: a43a3c62-ad6a-43cf-b149-30355203f773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:05:05,210-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64f725c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:05:05,213-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [38032122] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64f725c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:05:05,224-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [316ce5cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:05:05,225-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [316ce5cd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3703382a] Autorecovering 2 hosts >2018-05-29 13:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3703382a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:10:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a99d26c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26857632 >2018-05-29 13:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a99d26c] FINISH, SetVdsStatusVDSCommand, log id: 26857632 >2018-05-29 13:10:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4a99d26c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:10:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a99d26c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:10:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a8c021d >2018-05-29 13:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] FINISH, SetVdsStatusVDSCommand, log id: a8c021d >2018-05-29 13:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3eaab737 >2018-05-29 13:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3a8e1e0f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3eaab737 >2018-05-29 13:10:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8e1e0f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:10:02,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc90603] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5cca8bd6 >2018-05-29 13:10:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [dc90603] FINISH, GetHardwareInfoVDSCommand, log id: 5cca8bd6 >2018-05-29 13:10:02,975-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [343e0995] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:10:02,983-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1e4f73b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:10:02,989-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [11ee8734] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:10:02,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1e0e8d42] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 590002e7 >2018-05-29 13:10:03,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1e0e8d42] FINISH, GetHardwareInfoVDSCommand, log id: 590002e7 >2018-05-29 13:10:03,015-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4707db9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:10:03,024-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6164ba54] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:10:03,029-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [479798f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:10:03,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [479798f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73e2445e >2018-05-29 13:10:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [11ee8734] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d4c46e7 >2018-05-29 13:10:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:10:03,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [479798f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73e2445e >2018-05-29 13:10:03,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,420-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:10:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [11ee8734] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d4c46e7 >2018-05-29 13:10:03,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [479798f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,440-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [479798f0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:10:03,443-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [479798f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 479798f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:10:03,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:10:03,445-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [11ee8734] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:10:03,447-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [11ee8734] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:10:03,450-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [11ee8734] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 11ee8734, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:10:03,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 457491e0 >2018-05-29 13:10:03,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:10:03,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:10:03,545-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:10:03,545-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:10:03,545-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:10:03,545-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 457491e0 >2018-05-29 13:10:03,546-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:10:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56ac405b >2018-05-29 13:10:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:10:03,558-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [45d91327] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:10:03,560-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45d91327] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 653f413c >2018-05-29 13:10:03,562-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45d91327] FINISH, SetVdsStatusVDSCommand, log id: 653f413c >2018-05-29 13:10:03,570-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45d91327, Job ID: dd7c2a65-38f4-47ee-8c5c-c57bb2eeab08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:10:03,574-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6164ba54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:10:03,576-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45d91327] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6164ba54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:10:03,587-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2a5c0fa3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:10:03,588-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2a5c0fa3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:10:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:10:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:10:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56ac405b >2018-05-29 13:10:03,651-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:10:03,664-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [ad9c703] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:10:03,665-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [ad9c703] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b102443 >2018-05-29 13:10:03,667-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [ad9c703] FINISH, SetVdsStatusVDSCommand, log id: 1b102443 >2018-05-29 13:10:03,672-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ad9c703, Job ID: 5a7a3034-b0c0-4637-80e9-456671064c38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:10:03,677-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e4f73b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:10:03,679-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [ad9c703] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e4f73b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:10:03,689-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [613a8c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:10:03,690-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [613a8c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:15:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ff9aa2d] Autorecovering 2 hosts >2018-05-29 13:15:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1ff9aa2d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [17cf8043] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cad7c56 >2018-05-29 13:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [17cf8043] FINISH, SetVdsStatusVDSCommand, log id: 3cad7c56 >2018-05-29 13:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [17cf8043] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:15:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [17cf8043] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7271a2ea >2018-05-29 13:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] FINISH, SetVdsStatusVDSCommand, log id: 7271a2ea >2018-05-29 13:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a53653 >2018-05-29 13:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7ef27c21] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a53653 >2018-05-29 13:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7ef27c21] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:15:01,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77319510] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6865854f >2018-05-29 13:15:01,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [77319510] FINISH, GetHardwareInfoVDSCommand, log id: 6865854f >2018-05-29 13:15:01,339-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7eca4b80] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:15:01,348-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5afb0565] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:15:01,353-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [8f534c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:15:01,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [311d58fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1b407d96 >2018-05-29 13:15:01,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [8f534c6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a3a60fc >2018-05-29 13:15:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [311d58fc] FINISH, GetHardwareInfoVDSCommand, log id: 1b407d96 >2018-05-29 13:15:01,374-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5fb658a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:15:01,378-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [660b2bca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:15:01,383-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5e2427f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:15:01,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5e2427f8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 40d5bf74 >2018-05-29 13:15:01,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:15:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [8f534c6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a3a60fc >2018-05-29 13:15:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,746-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [8f534c6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,750-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [8f534c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:15:01,752-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [8f534c6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8f534c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:15:01,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13e0ea10 >2018-05-29 13:15:01,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:15:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:15:01,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5e2427f8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 40d5bf74 >2018-05-29 13:15:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,769-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:15:01,778-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5e2427f8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:15:01,781-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5e2427f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:15:01,784-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e2427f8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e2427f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:15:01,857-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:15:01,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:15:01,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:15:01,857-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:15:01,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13e0ea10 >2018-05-29 13:15:01,858-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:15:01,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 49df0673 >2018-05-29 13:15:01,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:15:01,870-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7085e96f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:15:01,872-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7085e96f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76e2131b >2018-05-29 13:15:01,874-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7085e96f] FINISH, SetVdsStatusVDSCommand, log id: 76e2131b >2018-05-29 13:15:01,905-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7085e96f, Job ID: 3fe0a810-e4eb-4690-bf29-9e12b32e9962, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:15:01,909-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5afb0565, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:15:01,912-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7085e96f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5afb0565, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:15:01,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [33423315] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:15:01,924-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [33423315] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:15:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:15:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:15:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 49df0673 >2018-05-29 13:15:01,961-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:15:01,974-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [d174964] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:15:01,976-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [d174964] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10a301be >2018-05-29 13:15:01,978-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [d174964] FINISH, SetVdsStatusVDSCommand, log id: 10a301be >2018-05-29 13:15:01,983-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: d174964, Job ID: 264d8f4c-710e-4de2-8be3-a2e0211f1929, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:15:01,988-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 660b2bca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:15:01,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d174964] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 660b2bca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:15:02,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [50de081c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:15:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [50de081c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:20:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [35e2ea4a] Autorecovering 2 hosts >2018-05-29 13:20:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [35e2ea4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:20:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:20:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:20:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6a6f654d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a48cc90 >2018-05-29 13:20:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6a6f654d] FINISH, SetVdsStatusVDSCommand, log id: 4a48cc90 >2018-05-29 13:20:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6a6f654d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:20:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6a6f654d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:20:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1336f75b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 879a272 >2018-05-29 13:20:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1336f75b] FINISH, SetVdsStatusVDSCommand, log id: 879a272 >2018-05-29 13:20:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1336f75b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6bb9c6de >2018-05-29 13:20:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1336f75b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6bb9c6de >2018-05-29 13:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1336f75b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:20:02,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [258a57f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7d3efbcb >2018-05-29 13:20:02,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [258a57f7] FINISH, GetHardwareInfoVDSCommand, log id: 7d3efbcb >2018-05-29 13:20:02,614-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [52ee24f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:20:02,622-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [56ed6b5a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:20:02,628-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [363f384e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:20:02,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4c3cd69c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 40c9be86 >2018-05-29 13:20:02,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4c3cd69c] FINISH, GetHardwareInfoVDSCommand, log id: 40c9be86 >2018-05-29 13:20:02,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [363f384e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c184893 >2018-05-29 13:20:02,651-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [447830c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:20:02,655-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [92a35c7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:20:02,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [17f5f7fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:20:02,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [17f5f7fc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59763160 >2018-05-29 13:20:03,019-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:20:03,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [363f384e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c184893 >2018-05-29 13:20:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,024-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,028-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,036-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,037-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [363f384e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,040-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [363f384e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:20:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:20:03,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [17f5f7fc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59763160 >2018-05-29 13:20:03,043-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [363f384e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 363f384e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:20:03,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,046-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e72834 >2018-05-29 13:20:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:20:03,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,051-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:20:03,056-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [17f5f7fc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:20:03,058-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [17f5f7fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:20:03,061-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [17f5f7fc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 17f5f7fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:20:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:20:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:20:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e72834 >2018-05-29 13:20:03,148-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:20:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f311b54 >2018-05-29 13:20:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:20:03,162-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [e1b50ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:20:03,163-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1b50ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5febb576 >2018-05-29 13:20:03,166-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1b50ca] FINISH, SetVdsStatusVDSCommand, log id: 5febb576 >2018-05-29 13:20:03,174-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: e1b50ca, Job ID: 8a220d70-03e1-42c1-98a2-b014253982ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:20:03,179-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 56ed6b5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:20:03,181-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [e1b50ca] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 56ed6b5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:20:03,192-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [182bf31a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:20:03,193-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [182bf31a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:20:03,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:20:03,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:20:03,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: f311b54 >2018-05-29 13:20:03,251-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:20:03,264-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6c896735] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:20:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6c896735] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7391fd2f >2018-05-29 13:20:03,267-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6c896735] FINISH, SetVdsStatusVDSCommand, log id: 7391fd2f >2018-05-29 13:20:03,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6c896735, Job ID: ace7b4c6-cd6e-454c-bf20-160af563101b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:20:03,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 92a35c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:20:03,279-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6c896735] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 92a35c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:20:03,289-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [783e2f06] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:20:03,290-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [783e2f06] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6770ff86] Autorecovering 2 hosts >2018-05-29 13:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6770ff86] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:25:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65223807] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 321d680d >2018-05-29 13:25:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65223807] FINISH, SetVdsStatusVDSCommand, log id: 321d680d >2018-05-29 13:25:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65223807] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:25:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [65223807] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68417c15] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31af719d >2018-05-29 13:25:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68417c15] FINISH, SetVdsStatusVDSCommand, log id: 31af719d >2018-05-29 13:25:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [68417c15] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43238b93 >2018-05-29 13:25:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [68417c15] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43238b93 >2018-05-29 13:25:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:25:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [68417c15] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:25:00,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [76e2082c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4774093d >2018-05-29 13:25:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [76e2082c] FINISH, GetHardwareInfoVDSCommand, log id: 4774093d >2018-05-29 13:25:00,859-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1b7611bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:25:00,868-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4fa7a4b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:25:00,873-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3d56cae5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:25:00,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3d56cae5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73518694 >2018-05-29 13:25:00,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3a278a54] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 666b32a0 >2018-05-29 13:25:00,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [3a278a54] FINISH, GetHardwareInfoVDSCommand, log id: 666b32a0 >2018-05-29 13:25:00,975-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [342d939b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:25:00,980-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7af93e49] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:25:00,985-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2f36cac1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:25:00,991-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2f36cac1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57c05565 >2018-05-29 13:25:01,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:25:01,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3d56cae5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73518694 >2018-05-29 13:25:01,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,273-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3d56cae5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,285-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3d56cae5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:25:01,288-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3d56cae5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d56cae5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:25:01,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 514096e7 >2018-05-29 13:25:01,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:25:01,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:25:01,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2f36cac1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57c05565 >2018-05-29 13:25:01,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,371-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,376-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:25:01,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2f36cac1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:25:01,383-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2f36cac1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:25:01,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2f36cac1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f36cac1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:25:01,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:25:01,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:25:01,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 514096e7 >2018-05-29 13:25:01,393-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:25:01,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55a72a9c >2018-05-29 13:25:01,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:25:01,409-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5c5adf35] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:25:01,410-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c5adf35] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47632cba >2018-05-29 13:25:01,412-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c5adf35] FINISH, SetVdsStatusVDSCommand, log id: 47632cba >2018-05-29 13:25:01,444-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c5adf35, Job ID: 02d4aaf5-3d18-4aef-87bd-4a951b9576c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:25:01,449-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4fa7a4b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:25:01,452-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5c5adf35] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4fa7a4b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:25:01,463-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [35e19c82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:25:01,464-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [35e19c82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:25:01,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:25:01,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:25:01,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55a72a9c >2018-05-29 13:25:01,499-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:25:01,512-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [59cd03dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:25:01,514-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [59cd03dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a6a6d94 >2018-05-29 13:25:01,516-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [59cd03dc] FINISH, SetVdsStatusVDSCommand, log id: 4a6a6d94 >2018-05-29 13:25:01,521-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59cd03dc, Job ID: 398f73b0-f703-4fdc-843d-6125c2386370, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:25:01,526-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7af93e49, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:25:01,528-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59cd03dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7af93e49, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:25:01,538-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3de85557] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:25:01,539-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3de85557] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [520d9247] Autorecovering 2 hosts >2018-05-29 13:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [520d9247] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c222b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67a24031 >2018-05-29 13:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c222b9b] FINISH, SetVdsStatusVDSCommand, log id: 67a24031 >2018-05-29 13:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c222b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:30:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5c222b9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d1961f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 251764c3 >2018-05-29 13:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [15d1961f] FINISH, SetVdsStatusVDSCommand, log id: 251764c3 >2018-05-29 13:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [15d1961f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 576d2722 >2018-05-29 13:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [15d1961f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 576d2722 >2018-05-29 13:30:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [15d1961f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:30:02,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [742c88ed] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 16a8c703 >2018-05-29 13:30:02,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [742c88ed] FINISH, GetHardwareInfoVDSCommand, log id: 16a8c703 >2018-05-29 13:30:02,140-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [552a4a99] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:30:02,148-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6a29f41d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:30:02,153-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6772a1b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:30:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6772a1b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f8447e6 >2018-05-29 13:30:02,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5d73035d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3c8ffde5 >2018-05-29 13:30:02,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5d73035d] FINISH, GetHardwareInfoVDSCommand, log id: 3c8ffde5 >2018-05-29 13:30:02,253-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6346b1b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:30:02,257-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [594499] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:30:02,262-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [788cd87a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:30:02,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [788cd87a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 507517b8 >2018-05-29 13:30:02,536-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:30:02,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6772a1b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f8447e6 >2018-05-29 13:30:02,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,540-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,544-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,545-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,549-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,549-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6772a1b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,552-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6772a1b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:30:02,555-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6772a1b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6772a1b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:30:02,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4aaf4fbc >2018-05-29 13:30:02,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:30:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:30:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [788cd87a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 507517b8 >2018-05-29 13:30:02,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:30:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [788cd87a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:30:02,650-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [788cd87a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:30:02,652-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [788cd87a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 788cd87a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:30:02,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:30:02,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:30:02,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4aaf4fbc >2018-05-29 13:30:02,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:30:02,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ead0bd2 >2018-05-29 13:30:02,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:30:02,672-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2954da65] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:30:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2954da65] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46ae5fc1 >2018-05-29 13:30:02,676-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2954da65] FINISH, SetVdsStatusVDSCommand, log id: 46ae5fc1 >2018-05-29 13:30:02,707-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2954da65, Job ID: 7f67a827-3902-474a-a49b-ea92b9e5eda3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:30:02,712-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a29f41d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:30:02,714-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2954da65] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a29f41d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:30:02,724-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5ee18fc1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:30:02,725-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5ee18fc1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:30:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:30:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:30:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ead0bd2 >2018-05-29 13:30:02,759-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:30:02,772-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [71c37238] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:30:02,774-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [71c37238] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16ada1c6 >2018-05-29 13:30:02,776-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [71c37238] FINISH, SetVdsStatusVDSCommand, log id: 16ada1c6 >2018-05-29 13:30:02,781-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71c37238, Job ID: f7c8cd9a-e810-497e-bcb8-2719c294f224, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:30:02,785-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 594499, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:30:02,787-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [71c37238] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 594499, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:30:02,798-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2ad8af3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:30:02,799-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2ad8af3d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [37b8e118] Autorecovering 2 hosts >2018-05-29 13:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [37b8e118] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:35:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [8199047] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 145815ad >2018-05-29 13:35:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [8199047] FINISH, SetVdsStatusVDSCommand, log id: 145815ad >2018-05-29 13:35:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [8199047] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:35:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [8199047] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:35:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:35:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:35:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91f3ebb >2018-05-29 13:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] FINISH, SetVdsStatusVDSCommand, log id: 91f3ebb >2018-05-29 13:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73bf2ade >2018-05-29 13:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [17fe7d70] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73bf2ade >2018-05-29 13:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:35:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [17fe7d70] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:35:00,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [727490d2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e1b9c7c >2018-05-29 13:35:00,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [727490d2] FINISH, GetHardwareInfoVDSCommand, log id: 3e1b9c7c >2018-05-29 13:35:00,327-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [46156d5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:35:00,331-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [38eb0ec5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:35:00,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5b4c840] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:35:00,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5b4c840] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5edc2f60 >2018-05-29 13:35:00,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [34c7385d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5cc7b76e >2018-05-29 13:35:00,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [34c7385d] FINISH, GetHardwareInfoVDSCommand, log id: 5cc7b76e >2018-05-29 13:35:00,453-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [16d82a01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:35:00,458-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2df56b84] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:35:00,463-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4e9c8989] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:35:00,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4e9c8989] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63f8347d >2018-05-29 13:35:00,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:35:00,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5b4c840] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5edc2f60 >2018-05-29 13:35:00,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,788-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5b4c840] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,796-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5b4c840] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:35:00,798-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b4c840] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b4c840, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:35:00,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e319e51 >2018-05-29 13:35:00,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:35:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:35:00,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4e9c8989] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63f8347d >2018-05-29 13:35:00,842-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,843-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,847-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:35:00,852-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4e9c8989] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:35:00,854-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4e9c8989] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:35:00,857-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4e9c8989] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e9c8989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:35:00,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:35:00,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:35:00,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e319e51 >2018-05-29 13:35:00,904-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:35:00,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44189cb6 >2018-05-29 13:35:00,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:35:00,917-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [59058a63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:35:00,918-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59058a63] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54613b78 >2018-05-29 13:35:00,920-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59058a63] FINISH, SetVdsStatusVDSCommand, log id: 54613b78 >2018-05-29 13:35:00,928-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 59058a63, Job ID: 51c06b48-47ac-48d0-ada3-9b497cdafbe3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:35:00,932-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 38eb0ec5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:35:00,934-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59058a63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 38eb0ec5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:35:00,945-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7ec3e1c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:35:00,946-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7ec3e1c2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:35:01,007-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:35:01,007-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:35:01,007-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44189cb6 >2018-05-29 13:35:01,007-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:35:01,019-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5997699] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:35:01,020-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5997699] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 344ce41a >2018-05-29 13:35:01,022-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5997699] FINISH, SetVdsStatusVDSCommand, log id: 344ce41a >2018-05-29 13:35:01,028-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5997699, Job ID: 332e1f25-9ea9-4887-b61f-a040f1a328c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:35:01,032-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2df56b84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:35:01,034-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5997699] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2df56b84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:35:01,044-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4b7574de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:35:01,045-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4b7574de] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [342aa372] Autorecovering 2 hosts >2018-05-29 13:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [342aa372] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [232993ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 717317da >2018-05-29 13:40:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [232993ed] FINISH, SetVdsStatusVDSCommand, log id: 717317da >2018-05-29 13:40:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [232993ed] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:40:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [232993ed] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:40:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [63e26b17] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75664c8 >2018-05-29 13:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [63e26b17] FINISH, SetVdsStatusVDSCommand, log id: 75664c8 >2018-05-29 13:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [63e26b17] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 727e2fcb >2018-05-29 13:40:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [63e26b17] FINISH, SetHaMaintenanceModeVDSCommand, log id: 727e2fcb >2018-05-29 13:40:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:40:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [63e26b17] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:40:01,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [292ffb79] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d340870 >2018-05-29 13:40:01,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [292ffb79] FINISH, GetHardwareInfoVDSCommand, log id: 4d340870 >2018-05-29 13:40:01,855-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [14ba6a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:40:01,864-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [18283ab1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:40:01,869-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2312b9e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:40:01,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2312b9e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 669fcd54 >2018-05-29 13:40:01,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5bec1198] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: f3d7dde >2018-05-29 13:40:01,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5bec1198] FINISH, GetHardwareInfoVDSCommand, log id: f3d7dde >2018-05-29 13:40:01,968-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [404711c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:40:01,973-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [449dc92d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:40:01,978-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d007af5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:40:01,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d007af5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 672909c8 >2018-05-29 13:40:02,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:40:02,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2312b9e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 669fcd54 >2018-05-29 13:40:02,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,256-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,266-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2312b9e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,270-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2312b9e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:40:02,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2312b9e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2312b9e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:40:02,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f5fd662 >2018-05-29 13:40:02,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:40:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:40:02,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d007af5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 672909c8 >2018-05-29 13:40:02,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,357-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:40:02,366-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d007af5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:40:02,370-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d007af5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:40:02,372-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d007af5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d007af5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:40:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:40:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:40:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f5fd662 >2018-05-29 13:40:02,375-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:40:02,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e0b2bea >2018-05-29 13:40:02,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:40:02,389-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2142209] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:40:02,391-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2142209] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 473ce6f3 >2018-05-29 13:40:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2142209] FINISH, SetVdsStatusVDSCommand, log id: 473ce6f3 >2018-05-29 13:40:02,424-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2142209, Job ID: 6b607832-b52f-4196-9ced-81deb4b25b43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:40:02,428-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18283ab1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:40:02,430-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2142209] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18283ab1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:40:02,442-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6271055d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:40:02,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6271055d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:40:02,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:40:02,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:40:02,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e0b2bea >2018-05-29 13:40:02,481-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:40:02,497-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3113295a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:40:02,498-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3113295a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35a72b55 >2018-05-29 13:40:02,501-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3113295a] FINISH, SetVdsStatusVDSCommand, log id: 35a72b55 >2018-05-29 13:40:02,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3113295a, Job ID: cfcdb6cd-8f34-4c4b-9b1e-4e96939ec9a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:40:02,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 449dc92d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:40:02,513-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3113295a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 449dc92d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:40:02,524-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3f6d6073] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:40:02,525-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3f6d6073] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ecf6b32] Autorecovering 2 hosts >2018-05-29 13:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1ecf6b32] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c8b0aa9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d7fbbad >2018-05-29 13:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c8b0aa9] FINISH, SetVdsStatusVDSCommand, log id: 7d7fbbad >2018-05-29 13:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7c8b0aa9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7c8b0aa9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8f1c78 >2018-05-29 13:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] FINISH, SetVdsStatusVDSCommand, log id: 8f1c78 >2018-05-29 13:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1c33ca9b >2018-05-29 13:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4780ba9d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1c33ca9b >2018-05-29 13:45:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4780ba9d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:45:03,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5382a076] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b3e6273 >2018-05-29 13:45:03,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5382a076] FINISH, GetHardwareInfoVDSCommand, log id: 2b3e6273 >2018-05-29 13:45:03,100-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7e138ea6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:45:03,108-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [715a2615] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:45:03,113-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [13ba5a71] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:45:03,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [13ba5a71] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 327768da >2018-05-29 13:45:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7fd25ff1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 375c2b23 >2018-05-29 13:45:03,258-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7fd25ff1] FINISH, GetHardwareInfoVDSCommand, log id: 375c2b23 >2018-05-29 13:45:03,270-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7e9ba8ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:45:03,275-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3454e54e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:45:03,280-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [210bd937] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:45:03,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [210bd937] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 354b231a >2018-05-29 13:45:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:45:03,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [13ba5a71] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 327768da >2018-05-29 13:45:03,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,511-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [13ba5a71] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,519-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [13ba5a71] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:45:03,521-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13ba5a71] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13ba5a71, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:45:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45c9bbe >2018-05-29 13:45:03,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:45:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:45:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:45:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45c9bbe >2018-05-29 13:45:03,626-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:45:03,639-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1fd5d347] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:45:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1fd5d347] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72c637bb >2018-05-29 13:45:03,643-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1fd5d347] FINISH, SetVdsStatusVDSCommand, log id: 72c637bb >2018-05-29 13:45:03,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:45:03,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [210bd937] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 354b231a >2018-05-29 13:45:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:45:03,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [210bd937] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:45:03,677-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [210bd937] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:45:03,677-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1fd5d347, Job ID: 248c3d75-0647-4ce4-977b-94befb89d9c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:45:03,679-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [210bd937] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 210bd937, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:45:03,683-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 715a2615, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:45:03,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23e12910 >2018-05-29 13:45:03,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:45:03,685-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fd5d347] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 715a2615, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:45:03,696-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4f08a629] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:45:03,698-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4f08a629] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:45:03,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:45:03,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:45:03,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23e12910 >2018-05-29 13:45:03,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:45:03,796-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4841bd82] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:45:03,797-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4841bd82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4bffaa02 >2018-05-29 13:45:03,799-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4841bd82] FINISH, SetVdsStatusVDSCommand, log id: 4bffaa02 >2018-05-29 13:45:03,831-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4841bd82, Job ID: ab04b836-3c6a-4bbb-97a3-a5a299c8ead3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:45:03,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3454e54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:45:03,837-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4841bd82] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3454e54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:45:03,847-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [63283a12] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:45:03,848-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [63283a12] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2eacf3cc] Autorecovering 2 hosts >2018-05-29 13:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2eacf3cc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [37384b38] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54559d3a >2018-05-29 13:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [37384b38] FINISH, SetVdsStatusVDSCommand, log id: 54559d3a >2018-05-29 13:50:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [37384b38] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:50:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [37384b38] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:50:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 413f7782 >2018-05-29 13:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] FINISH, SetVdsStatusVDSCommand, log id: 413f7782 >2018-05-29 13:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52b291e7 >2018-05-29 13:50:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [14cc66bf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52b291e7 >2018-05-29 13:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:50:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [14cc66bf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:50:01,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [37688b44] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 581a1477 >2018-05-29 13:50:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [37688b44] FINISH, GetHardwareInfoVDSCommand, log id: 581a1477 >2018-05-29 13:50:01,321-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4370eb9d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:50:01,329-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7acc00de] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:50:01,334-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7e584c54] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:50:01,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7e584c54] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ea38e30 >2018-05-29 13:50:01,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e249911] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73efd034 >2018-05-29 13:50:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e249911] FINISH, GetHardwareInfoVDSCommand, log id: 73efd034 >2018-05-29 13:50:01,445-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [430d3830] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:50:01,454-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3fd5027] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:50:01,459-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [194cdb7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:50:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [194cdb7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dce7166 >2018-05-29 13:50:01,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:50:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7e584c54] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ea38e30 >2018-05-29 13:50:01,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:01,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:01,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:01,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:01,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7e584c54] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:01,744-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7e584c54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:50:01,746-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7e584c54] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e584c54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:50:01,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61e42f30 >2018-05-29 13:50:01,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:50:01,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:50:01,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:50:01,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61e42f30 >2018-05-29 13:50:01,851-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:50:01,864-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [109d56b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:50:01,866-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [109d56b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78bd7680 >2018-05-29 13:50:01,868-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [109d56b0] FINISH, SetVdsStatusVDSCommand, log id: 78bd7680 >2018-05-29 13:50:01,900-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 109d56b0, Job ID: 459ca66f-6c95-4b92-a77c-f785af63c294, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:50:01,905-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7acc00de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:50:01,907-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [109d56b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7acc00de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:50:01,918-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [b7e1086] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:50:01,919-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [b7e1086] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:50:02,879-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:50:02,879-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [194cdb7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dce7166 >2018-05-29 13:50:02,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:02,883-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:02,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:02,888-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:02,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:50:02,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [194cdb7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:50:02,895-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [194cdb7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:50:02,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [194cdb7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 194cdb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:50:02,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23d79c8 >2018-05-29 13:50:02,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23d79c8 >2018-05-29 13:50:02,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:50:03,010-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [40d1f2cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:50:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [40d1f2cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26910a6 >2018-05-29 13:50:03,014-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [40d1f2cb] FINISH, SetVdsStatusVDSCommand, log id: 26910a6 >2018-05-29 13:50:03,019-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40d1f2cb, Job ID: 73388a54-ab2e-4870-8522-c2cadd025afc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:50:03,024-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fd5027, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:50:03,027-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [40d1f2cb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fd5027, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:50:03,037-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [27edf4b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:50:03,038-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [27edf4b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:54:38,349-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [12308afc] Backup check started. >2018-05-29 13:54:38,362-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12308afc] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption. >2018-05-29 13:54:38,362-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [12308afc] Backup check completed. >2018-05-29 13:54:43,030-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [29b2ecac] Attempting to update VMs/Templates Ovf. >2018-05-29 13:54:43,032-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 13:54:43,032-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 13:54:43,033-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:54:43,037-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 13:54:43,041-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 13:54:43,041-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 13:54:43,041-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 13:54:43,041-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 13:54:43,041-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 13:54:43,042-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [1cad24f4] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 13:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7baed0bf] Autorecovering 2 hosts >2018-05-29 13:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7baed0bf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5838a574] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74faa4a7 >2018-05-29 13:55:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5838a574] FINISH, SetVdsStatusVDSCommand, log id: 74faa4a7 >2018-05-29 13:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5838a574] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 13:55:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5838a574] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:55:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 13:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62482b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d5b0752 >2018-05-29 13:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [62482b1] FINISH, SetVdsStatusVDSCommand, log id: 3d5b0752 >2018-05-29 13:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62482b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 17e0a4ba >2018-05-29 13:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [62482b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 17e0a4ba >2018-05-29 13:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 13:55:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [62482b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 13:55:00,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5d682b65] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f7f3390 >2018-05-29 13:55:00,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5d682b65] FINISH, GetHardwareInfoVDSCommand, log id: 7f7f3390 >2018-05-29 13:55:00,685-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3d8b8012] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:55:00,693-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [3077ace5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:55:00,698-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3b77bff7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:55:00,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3b77bff7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e064557 >2018-05-29 13:55:01,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:55:01,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3b77bff7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e064557 >2018-05-29 13:55:01,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:01,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:01,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:01,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:01,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:01,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3b77bff7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:01,093-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3b77bff7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:55:01,096-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3b77bff7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b77bff7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:55:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1476fb0d >2018-05-29 13:55:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:55:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:55:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:55:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1476fb0d >2018-05-29 13:55:01,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:55:01,212-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2c8d76a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:55:01,213-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c8d76a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 227b874 >2018-05-29 13:55:01,215-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c8d76a3] FINISH, SetVdsStatusVDSCommand, log id: 227b874 >2018-05-29 13:55:01,249-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c8d76a3, Job ID: 583a1db1-7561-46a9-85d1-b8bb57c2372c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:55:01,254-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3077ace5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:55:01,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2c8d76a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3077ace5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:55:01,266-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [12743693] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 13:55:01,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [12743693] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 13:55:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a34815] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3dc1a1dd >2018-05-29 13:55:02,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a34815] FINISH, GetHardwareInfoVDSCommand, log id: 3dc1a1dd >2018-05-29 13:55:02,629-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3502eee7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:55:02,637-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7b680b65] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:55:02,642-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3a0e1842] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 13:55:02,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3a0e1842] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 342f83ab >2018-05-29 13:55:03,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 13:55:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [3a0e1842] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 342f83ab >2018-05-29 13:55:03,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:03,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:03,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 13:55:03,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [3a0e1842] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 13:55:03,050-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [3a0e1842] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 13:55:03,053-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3a0e1842] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a0e1842, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 13:55:03,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d24545d >2018-05-29 13:55:03,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:55:03,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 13:55:03,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 13:55:03,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d24545d >2018-05-29 13:55:03,161-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 13:55:03,174-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6833fcad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:55:03,175-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6833fcad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 671ff883 >2018-05-29 13:55:03,177-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6833fcad] FINISH, SetVdsStatusVDSCommand, log id: 671ff883 >2018-05-29 13:55:03,208-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6833fcad, Job ID: 197a579b-ce3b-4c2c-b709-abdba0934c46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 13:55:03,213-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b680b65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 13:55:03,215-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6833fcad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b680b65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 13:55:03,226-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [58bbbd12] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 13:55:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [58bbbd12] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d0a674] Autorecovering 2 hosts >2018-05-29 14:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1d0a674] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2debd8f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18d1a1f9 >2018-05-29 14:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2debd8f6] FINISH, SetVdsStatusVDSCommand, log id: 18d1a1f9 >2018-05-29 14:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2debd8f6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2debd8f6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [498aa64d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 384b5bc4 >2018-05-29 14:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [498aa64d] FINISH, SetVdsStatusVDSCommand, log id: 384b5bc4 >2018-05-29 14:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [498aa64d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a70e268 >2018-05-29 14:00:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [498aa64d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a70e268 >2018-05-29 14:00:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:00:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [498aa64d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:00:00,862-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [78c1e55c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413ba3ff >2018-05-29 14:00:00,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [78c1e55c] FINISH, GetHardwareInfoVDSCommand, log id: 413ba3ff >2018-05-29 14:00:00,876-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [493e45db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:00:00,885-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [191c659e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:00:00,890-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1369d130] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:00:00,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1369d130] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6357682f >2018-05-29 14:00:01,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:00:01,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1369d130] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6357682f >2018-05-29 14:00:01,286-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:01,287-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:01,290-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:01,291-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:01,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:01,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1369d130] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:01,298-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1369d130] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:00:01,301-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1369d130] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1369d130, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:00:01,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 155cd37e >2018-05-29 14:00:01,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:00:01,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:00:01,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:00:01,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 155cd37e >2018-05-29 14:00:01,403-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:00:01,415-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [37ce6bf7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:00:01,417-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [37ce6bf7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ad9fede >2018-05-29 14:00:01,419-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [37ce6bf7] FINISH, SetVdsStatusVDSCommand, log id: 3ad9fede >2018-05-29 14:00:01,427-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 37ce6bf7, Job ID: a1e60886-4698-44df-bca2-a7b53426faae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:00:01,431-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 191c659e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:00:01,433-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [37ce6bf7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 191c659e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:00:01,444-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [21da4410] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:00:01,445-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [21da4410] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:00:02,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1127ed4d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 67019985 >2018-05-29 14:00:02,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1127ed4d] FINISH, GetHardwareInfoVDSCommand, log id: 67019985 >2018-05-29 14:00:02,160-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [37d7897a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:00:02,164-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [687f02d2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:00:02,169-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [26fc55bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:00:02,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [26fc55bc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 608dec95 >2018-05-29 14:00:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:00:03,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [26fc55bc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 608dec95 >2018-05-29 14:00:03,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:03,600-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:03,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:03,605-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:03,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:00:03,609-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [26fc55bc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:00:03,612-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [26fc55bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:00:03,615-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26fc55bc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 26fc55bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:00:03,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: db5a90a >2018-05-29 14:00:03,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:00:03,718-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:00:03,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:00:03,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:00:03,718-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:00:03,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: db5a90a >2018-05-29 14:00:03,719-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:00:03,732-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [ab9e32] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:00:03,733-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ab9e32] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f457028 >2018-05-29 14:00:03,735-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ab9e32] FINISH, SetVdsStatusVDSCommand, log id: 6f457028 >2018-05-29 14:00:03,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ab9e32, Job ID: 7572ebb5-6bc6-42d9-b45f-a37d080cc583, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:00:03,770-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 687f02d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:00:03,772-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ab9e32] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 687f02d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:00:03,783-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [23d492d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:00:03,784-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [23d492d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b26f55] Autorecovering 2 hosts >2018-05-29 14:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b26f55] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17f1f870] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 334eba43 >2018-05-29 14:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17f1f870] FINISH, SetVdsStatusVDSCommand, log id: 334eba43 >2018-05-29 14:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17f1f870] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [17f1f870] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74850fff >2018-05-29 14:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] FINISH, SetVdsStatusVDSCommand, log id: 74850fff >2018-05-29 14:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1fff7f90 >2018-05-29 14:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [39aff3f0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1fff7f90 >2018-05-29 14:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [39aff3f0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:05:01,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d52265a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 39f58d4b >2018-05-29 14:05:01,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d52265a] FINISH, GetHardwareInfoVDSCommand, log id: 39f58d4b >2018-05-29 14:05:01,523-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [c6e919e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:05:01,532-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2815d724] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:05:01,537-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [658dec40] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:05:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [658dec40] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b6b560e >2018-05-29 14:05:01,925-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:05:01,925-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [658dec40] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b6b560e >2018-05-29 14:05:01,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:01,929-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:01,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:01,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:01,938-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [658dec40] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:01,942-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [658dec40] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:05:01,944-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [658dec40] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 658dec40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:05:01,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d0cc176 >2018-05-29 14:05:01,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:05:02,045-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2ab124d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44845f84 >2018-05-29 14:05:02,046-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [2ab124d4] FINISH, GetHardwareInfoVDSCommand, log id: 44845f84 >2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:05:02,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:05:02,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:05:02,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d0cc176 >2018-05-29 14:05:02,050-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:05:02,060-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [c70f805] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:05:02,064-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [34b13057] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:05:02,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b13057] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454b505f >2018-05-29 14:05:02,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b13057] FINISH, SetVdsStatusVDSCommand, log id: 454b505f >2018-05-29 14:05:02,069-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2e86b6e2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:05:02,075-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [342a2d6a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:05:02,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [342a2d6a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23c5bbd6 >2018-05-29 14:05:02,100-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34b13057, Job ID: 88a822ac-5091-46cf-9bac-2711c71b905a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:05:02,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2815d724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:05:02,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [34b13057] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2815d724, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:05:02,118-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [cb858c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:05:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [cb858c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:05:02,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:05:02,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [342a2d6a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23c5bbd6 >2018-05-29 14:05:02,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:02,472-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:02,476-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:02,476-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:02,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:05:02,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [342a2d6a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:05:02,484-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [342a2d6a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:05:02,487-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [342a2d6a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 342a2d6a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:05:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6035c247 >2018-05-29 14:05:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:05:02,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:05:02,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:05:02,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:05:02,591-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:05:02,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6035c247 >2018-05-29 14:05:02,591-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:05:02,603-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [332efd95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:05:02,605-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [332efd95] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12761e93 >2018-05-29 14:05:02,607-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [332efd95] FINISH, SetVdsStatusVDSCommand, log id: 12761e93 >2018-05-29 14:05:02,612-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 332efd95, Job ID: 46d150ad-644b-4e41-9ff9-62dfd3f9d88b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:05:02,616-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e86b6e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:05:02,619-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [332efd95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e86b6e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:05:02,629-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [140dbee1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:05:02,630-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [140dbee1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:09:38,620-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-1) [393dc732] Connected to host dell-per630-06.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:a1hh1jl16Nd9hTtuA/TUOOzVFcsyda1SCcq54arWb6Y >2018-05-29 14:09:38,624-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [1050a6a8] Connected to host dell-per630-07.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:Li7+tEWpDe9k+6GYLAsCWnLuHH60ubhmHCZCQdQb26g >2018-05-29 14:09:38,627-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [53fae673] Connected to host dell-per630-05.css.lab.eng.rdu2.redhat.com with SSH key fingerprint: SHA256:/j16in72Hste2xxE4lLGG0zCeN8aAMgKVf8TFwHlL7I >2018-05-29 14:09:38,700-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-1) [393dc732] Installation of dell-per630-06.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2018-05-29 14:09:38,700-04 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-7-thread-1) [393dc732] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh >2018-05-29 14:09:38,703-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [1050a6a8] Installation of dell-per630-07.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2018-05-29 14:09:38,703-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-3) [1050a6a8] SSH execute 'root@dell-per630-07.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2018-05-29 14:09:38,706-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [53fae673] Installation of dell-per630-05.css.lab.eng.rdu2.redhat.com. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2018-05-29 14:09:38,706-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-5) [53fae673] SSH execute 'root@dell-per630-05.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2018-05-29 14:09:38,714-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-1) [393dc732] SSH execute 'root@dell-per630-06.css.lab.eng.rdu2.redhat.com' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2018-05-29 14:09:39,738-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Initializing >2018-05-29 14:09:39,745-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment setup >2018-05-29 14:09:39,756-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Initializing >2018-05-29 14:09:39,756-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment packages setup >2018-05-29 14:09:39,758-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Initializing >2018-05-29 14:09:39,762-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment setup >2018-05-29 14:09:39,764-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment setup >2018-05-29 14:09:39,773-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment packages setup >2018-05-29 14:09:39,774-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment packages setup >2018-05-29 14:09:40,349-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Programs detection >2018-05-29 14:09:40,361-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Programs detection >2018-05-29 14:09:40,367-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Environment customization >2018-05-29 14:09:40,368-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Programs detection >2018-05-29 14:09:40,378-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-5b44yl.log' >2018-05-29 14:09:40,379-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Environment customization >2018-05-29 14:09:40,385-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Environment customization >2018-05-29 14:09:40,389-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-iimpip.log' >2018-05-29 14:09:40,396-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Logs at host located at: '/tmp/ovirt-host-mgmt-20180529140942-7mruz0.log' >2018-05-29 14:09:40,397-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Setup validation >2018-05-29 14:09:40,405-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Setup validation >2018-05-29 14:09:40,408-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Transaction setup >2018-05-29 14:09:40,418-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Transaction setup >2018-05-29 14:09:40,426-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Setup validation >2018-05-29 14:09:40,438-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Transaction setup >2018-05-29 14:09:40,699-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Misc configuration >2018-05-29 14:09:40,700-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Package installation >2018-05-29 14:09:40,714-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Misc configuration >2018-05-29 14:09:40,714-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Package installation >2018-05-29 14:09:40,725-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Misc configuration >2018-05-29 14:09:40,726-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Package installation >2018-05-29 14:09:51,732-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Misc configuration >2018-05-29 14:09:51,734-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Transaction commit >2018-05-29 14:09:51,776-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Misc configuration >2018-05-29 14:09:51,779-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Transaction commit >2018-05-29 14:09:51,779-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Misc configuration >2018-05-29 14:09:51,781-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Transaction commit >2018-05-29 14:09:52,037-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Closing up >2018-05-29 14:09:52,039-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Pre-termination >2018-05-29 14:09:52,048-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-07.css.lab.eng.rdu2.redhat.com-1050a6a8.log' >2018-05-29 14:09:52,086-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Closing up >2018-05-29 14:09:52,087-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Closing up >2018-05-29 14:09:52,088-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Pre-termination >2018-05-29 14:09:52,089-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Pre-termination >2018-05-29 14:09:52,099-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-06.css.lab.eng.rdu2.redhat.com-393dc732.log' >2018-05-29 14:09:52,100-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20180529140952-dell-per630-05.css.lab.eng.rdu2.redhat.com-53fae673.log' >2018-05-29 14:09:52,104-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [1050a6a8] Stage: Termination >2018-05-29 14:09:52,149-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-3) [1050a6a8] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-07.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'. >2018-05-29 14:09:52,159-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [53fae673] Stage: Termination >2018-05-29 14:09:52,159-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [393dc732] Stage: Termination >2018-05-29 14:09:52,201-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-1) [393dc732] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-06.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'. >2018-05-29 14:09:52,212-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-5) [53fae673] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host dell-per630-05.css.lab.eng.rdu2.redhat.com was completed successfully with message 'no updates found.'. >2018-05-29 14:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [58e7dfc0] Autorecovering 2 hosts >2018-05-29 14:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [58e7dfc0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6dc41352] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2baac71 >2018-05-29 14:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6dc41352] FINISH, SetVdsStatusVDSCommand, log id: 2baac71 >2018-05-29 14:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6dc41352] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6dc41352] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cc0fbb2 >2018-05-29 14:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] FINISH, SetVdsStatusVDSCommand, log id: cc0fbb2 >2018-05-29 14:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4c2b82dc >2018-05-29 14:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5c0c2f62] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4c2b82dc >2018-05-29 14:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:10:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5c0c2f62] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:10:00,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3d7a3c0b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50e1481a >2018-05-29 14:10:00,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3d7a3c0b] FINISH, GetHardwareInfoVDSCommand, log id: 50e1481a >2018-05-29 14:10:00,266-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6dc56e4d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:10:00,275-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4dfd790f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:10:00,280-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [320ea1b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:10:00,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [320ea1b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 46363e0c >2018-05-29 14:10:00,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:10:00,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [320ea1b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 46363e0c >2018-05-29 14:10:00,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:00,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:00,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:00,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [320ea1b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:00,691-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [320ea1b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:10:00,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [320ea1b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 320ea1b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:10:00,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 42cac827 >2018-05-29 14:10:00,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:10:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:10:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:10:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 42cac827 >2018-05-29 14:10:00,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:10:00,812-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [668ee421] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:10:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [668ee421] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 519a5f34 >2018-05-29 14:10:00,815-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [668ee421] FINISH, SetVdsStatusVDSCommand, log id: 519a5f34 >2018-05-29 14:10:00,844-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 668ee421, Job ID: a26d7e58-6ccd-49bf-98df-d6d2c2739034, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:10:00,848-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4dfd790f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:10:00,850-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [668ee421] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4dfd790f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:10:00,861-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [14ac7a58] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:10:00,862-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [14ac7a58] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:10:02,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [12d2e1bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 10b2dca2 >2018-05-29 14:10:02,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [12d2e1bf] FINISH, GetHardwareInfoVDSCommand, log id: 10b2dca2 >2018-05-29 14:10:02,902-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [548d2653] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:10:02,906-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3b23e9da] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:10:02,911-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [637186ce] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:10:02,918-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [637186ce] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f76eb0c >2018-05-29 14:10:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:10:03,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [637186ce] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f76eb0c >2018-05-29 14:10:03,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:03,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:10:03,298-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [637186ce] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:10:03,300-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [637186ce] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:10:03,303-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [637186ce] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 637186ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:10:03,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13e0063f >2018-05-29 14:10:03,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:10:03,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:10:03,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:10:03,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13e0063f >2018-05-29 14:10:03,408-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:10:03,421-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f28a8ea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:10:03,422-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f28a8ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14854bfe >2018-05-29 14:10:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f28a8ea] FINISH, SetVdsStatusVDSCommand, log id: 14854bfe >2018-05-29 14:10:03,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f28a8ea, Job ID: d1edd705-16e1-4ca9-bdde-afc5f0123890, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:10:03,462-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b23e9da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:10:03,464-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f28a8ea] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b23e9da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:10:03,475-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [348e2f3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:10:03,476-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [348e2f3a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a202684] Autorecovering 2 hosts >2018-05-29 14:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a202684] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18bf0693] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 403d94a6 >2018-05-29 14:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [18bf0693] FINISH, SetVdsStatusVDSCommand, log id: 403d94a6 >2018-05-29 14:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [18bf0693] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:15:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [18bf0693] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2d22c176] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21709b7c >2018-05-29 14:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2d22c176] FINISH, SetVdsStatusVDSCommand, log id: 21709b7c >2018-05-29 14:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2d22c176] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 678deffc >2018-05-29 14:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2d22c176] FINISH, SetHaMaintenanceModeVDSCommand, log id: 678deffc >2018-05-29 14:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2d22c176] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:15:01,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a473b5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6aafcf5d >2018-05-29 14:15:01,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33a473b5] FINISH, GetHardwareInfoVDSCommand, log id: 6aafcf5d >2018-05-29 14:15:01,179-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [66602826] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:15:01,187-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6a5f6ed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:15:01,192-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [492cf03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:15:01,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [492cf03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51527c33 >2018-05-29 14:15:01,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [205803a1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6445facc >2018-05-29 14:15:01,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [205803a1] FINISH, GetHardwareInfoVDSCommand, log id: 6445facc >2018-05-29 14:15:01,472-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1a5977ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:15:01,476-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3d927f47] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:15:01,481-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [30d6c0ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:15:01,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [30d6c0ab] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 292bae1e >2018-05-29 14:15:01,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:15:01,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [492cf03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51527c33 >2018-05-29 14:15:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,589-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [492cf03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,591-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [492cf03] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:15:01,594-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [492cf03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 492cf03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:15:01,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4416eed >2018-05-29 14:15:01,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:15:01,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:15:01,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:15:01,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4416eed >2018-05-29 14:15:01,698-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:15:01,711-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [243823f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:15:01,713-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [243823f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dcbe46b >2018-05-29 14:15:01,715-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [243823f6] FINISH, SetVdsStatusVDSCommand, log id: 2dcbe46b >2018-05-29 14:15:01,723-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 243823f6, Job ID: 4d271992-1b41-4fa5-b469-54be081d8e3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:15:01,728-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a5f6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:15:01,730-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [243823f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a5f6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:15:01,740-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4a8bd8c5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:15:01,741-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4a8bd8c5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:15:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:15:01,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [30d6c0ab] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 292bae1e >2018-05-29 14:15:01,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,877-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,882-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:15:01,886-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [30d6c0ab] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:15:01,889-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [30d6c0ab] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:15:01,892-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30d6c0ab] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30d6c0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:15:01,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 32521552 >2018-05-29 14:15:01,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:15:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:15:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:15:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 32521552 >2018-05-29 14:15:02,001-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:15:02,013-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [70b25635] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:15:02,015-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70b25635] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 531be07c >2018-05-29 14:15:02,017-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70b25635] FINISH, SetVdsStatusVDSCommand, log id: 531be07c >2018-05-29 14:15:02,022-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 70b25635, Job ID: 2c8b197c-62d2-48b9-bfb6-3e62cfa8e225, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:15:02,027-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d927f47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:15:02,029-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [70b25635] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d927f47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:15:02,039-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4f690dd6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:15:02,041-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4f690dd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3727c455] Autorecovering 2 hosts >2018-05-29 14:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3727c455] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:20:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:20:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:20:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [171bac7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bbea172 >2018-05-29 14:20:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [171bac7] FINISH, SetVdsStatusVDSCommand, log id: bbea172 >2018-05-29 14:20:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [171bac7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:20:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [171bac7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3277baf1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47c6af9e >2018-05-29 14:20:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3277baf1] FINISH, SetVdsStatusVDSCommand, log id: 47c6af9e >2018-05-29 14:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3277baf1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c9b81d1 >2018-05-29 14:20:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [3277baf1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c9b81d1 >2018-05-29 14:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:20:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3277baf1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:20:02,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20a5b307] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 742fe286 >2018-05-29 14:20:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20a5b307] FINISH, GetHardwareInfoVDSCommand, log id: 742fe286 >2018-05-29 14:20:02,771-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [f16c3c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:20:02,780-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6bf88d1a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:20:02,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12af08bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:20:02,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12af08bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a64d31b >2018-05-29 14:20:02,915-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5aecfec7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 320b757b >2018-05-29 14:20:02,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5aecfec7] FINISH, GetHardwareInfoVDSCommand, log id: 320b757b >2018-05-29 14:20:02,928-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [39ebc155] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:20:02,933-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5f4fad2e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:20:02,938-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [170be02b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:20:02,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [170be02b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79b3d285 >2018-05-29 14:20:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:20:03,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [170be02b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79b3d285 >2018-05-29 14:20:03,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:03,331-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:03,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:03,336-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:03,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:03,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [170be02b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:03,343-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [170be02b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:20:03,345-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [170be02b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 170be02b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:20:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 273df5bb >2018-05-29 14:20:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:20:03,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:20:03,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:20:03,447-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 273df5bb >2018-05-29 14:20:03,447-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:20:03,460-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [60bcb125] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:20:03,462-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60bcb125] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5dd7e83c >2018-05-29 14:20:03,463-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [60bcb125] FINISH, SetVdsStatusVDSCommand, log id: 5dd7e83c >2018-05-29 14:20:03,492-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60bcb125, Job ID: ce7de919-155d-4ca7-82a0-55247d933912, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:20:03,497-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f4fad2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:20:03,499-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [60bcb125] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f4fad2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:20:03,509-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [66470f69] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:20:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [66470f69] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:20:04,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:20:04,165-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12af08bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a64d31b >2018-05-29 14:20:04,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:04,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:04,174-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:04,175-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:04,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:20:04,179-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12af08bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:20:04,182-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12af08bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:20:04,185-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12af08bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12af08bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:20:04,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3700bf46 >2018-05-29 14:20:04,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:20:04,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:20:04,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:20:04,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3700bf46 >2018-05-29 14:20:04,287-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:20:04,299-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [cae50b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:20:04,300-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [cae50b9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69861115 >2018-05-29 14:20:04,302-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [cae50b9] FINISH, SetVdsStatusVDSCommand, log id: 69861115 >2018-05-29 14:20:04,308-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: cae50b9, Job ID: 8363ee7e-e818-4e10-b432-fae9b328ad06, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:20:04,312-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6bf88d1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:20:04,314-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cae50b9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6bf88d1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:20:04,325-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [481de46f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:20:04,326-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [481de46f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d275340] Autorecovering 2 hosts >2018-05-29 14:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2d275340] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:25:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29631fb2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 875d59 >2018-05-29 14:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29631fb2] FINISH, SetVdsStatusVDSCommand, log id: 875d59 >2018-05-29 14:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29631fb2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:25:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [29631fb2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58938c55] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e8c502 >2018-05-29 14:25:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58938c55] FINISH, SetVdsStatusVDSCommand, log id: 6e8c502 >2018-05-29 14:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58938c55] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 75fdb3b7 >2018-05-29 14:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58938c55] FINISH, SetHaMaintenanceModeVDSCommand, log id: 75fdb3b7 >2018-05-29 14:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:25:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58938c55] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:25:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [369680fa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 493c3800 >2018-05-29 14:25:01,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [369680fa] FINISH, GetHardwareInfoVDSCommand, log id: 493c3800 >2018-05-29 14:25:01,085-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6b227e75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:25:01,093-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [711c75ca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:25:01,098-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5b5f80f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:25:01,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5b5f80f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7544b9f5 >2018-05-29 14:25:01,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:25:01,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5b5f80f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7544b9f5 >2018-05-29 14:25:01,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:01,498-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:01,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:01,503-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:01,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5b5f80f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:01,510-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5b5f80f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:25:01,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5b5f80f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b5f80f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:25:01,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66b92ab5 >2018-05-29 14:25:01,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:25:01,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:25:01,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:25:01,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66b92ab5 >2018-05-29 14:25:01,621-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:25:01,634-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4a8120fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:25:01,635-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4a8120fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e146537 >2018-05-29 14:25:01,638-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4a8120fd] FINISH, SetVdsStatusVDSCommand, log id: 3e146537 >2018-05-29 14:25:01,669-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a8120fd, Job ID: f6c250f9-aefe-46df-87f8-309187741c40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:25:01,674-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 711c75ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:25:01,676-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4a8120fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 711c75ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:25:01,687-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [30574dd4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:25:01,688-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [30574dd4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:25:02,046-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [16be73e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 62b16eed >2018-05-29 14:25:02,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [16be73e6] FINISH, GetHardwareInfoVDSCommand, log id: 62b16eed >2018-05-29 14:25:02,060-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5211e126] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:25:02,068-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [105a3ead] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:25:02,073-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c37ae51] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:25:02,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c37ae51] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 590c24f6 >2018-05-29 14:25:02,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:25:02,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c37ae51] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 590c24f6 >2018-05-29 14:25:02,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:02,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:02,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:02,470-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:25:02,475-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c37ae51] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:25:02,478-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c37ae51] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:25:02,480-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c37ae51] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c37ae51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:25:02,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38944f92 >2018-05-29 14:25:02,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:25:02,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:25:02,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:25:02,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38944f92 >2018-05-29 14:25:02,586-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:25:02,600-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3ef11697] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:25:02,601-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ef11697] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39af0a9b >2018-05-29 14:25:02,603-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3ef11697] FINISH, SetVdsStatusVDSCommand, log id: 39af0a9b >2018-05-29 14:25:02,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ef11697, Job ID: 756ad5ba-1a51-4007-a981-76888612dfe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:25:02,641-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 105a3ead, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:25:02,643-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3ef11697] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 105a3ead, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:25:02,654-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [5fdfea96] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:25:02,655-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5fdfea96] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3c63f86e] Autorecovering 2 hosts >2018-05-29 14:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3c63f86e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:30:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:30:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:30:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [505cf410] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2aa5cb41 >2018-05-29 14:30:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [505cf410] FINISH, SetVdsStatusVDSCommand, log id: 2aa5cb41 >2018-05-29 14:30:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [505cf410] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:30:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [505cf410] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:30:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:30:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:30:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47649797 >2018-05-29 14:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] FINISH, SetVdsStatusVDSCommand, log id: 47649797 >2018-05-29 14:30:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 612f2126 >2018-05-29 14:30:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [45fe30d2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 612f2126 >2018-05-29 14:30:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:30:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [45fe30d2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:30:00,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [13d5905e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5628f673 >2018-05-29 14:30:00,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [13d5905e] FINISH, GetHardwareInfoVDSCommand, log id: 5628f673 >2018-05-29 14:30:00,380-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [4b6c88d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:30:00,389-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4932a413] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:30:00,394-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [27a5d96b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:30:00,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [27a5d96b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 769286ad >2018-05-29 14:30:00,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:30:00,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [27a5d96b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 769286ad >2018-05-29 14:30:00,791-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:00,792-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:00,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:00,796-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:00,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:00,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [27a5d96b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:00,804-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [27a5d96b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:30:00,806-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27a5d96b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27a5d96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:30:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64f17ad6 >2018-05-29 14:30:00,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:30:00,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:30:00,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:30:00,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64f17ad6 >2018-05-29 14:30:00,912-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:30:00,925-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4cd8f0d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:30:00,926-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4cd8f0d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ad5136e >2018-05-29 14:30:00,928-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4cd8f0d6] FINISH, SetVdsStatusVDSCommand, log id: 7ad5136e >2018-05-29 14:30:00,960-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4cd8f0d6, Job ID: 04bb2714-7397-413a-96b5-d910f85e7cc8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:30:00,964-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4932a413, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:30:00,966-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4cd8f0d6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4932a413, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:30:00,977-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3b701538] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:30:00,978-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3b701538] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:30:02,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [d94a14b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6bba92a6 >2018-05-29 14:30:02,234-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [d94a14b] FINISH, GetHardwareInfoVDSCommand, log id: 6bba92a6 >2018-05-29 14:30:02,246-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [52836f3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:30:02,255-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [29903996] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:30:02,260-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [53659830] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:30:02,271-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [53659830] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7264cb6c >2018-05-29 14:30:02,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:30:02,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [53659830] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7264cb6c >2018-05-29 14:30:02,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:02,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:02,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:02,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:02,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:30:02,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [53659830] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:30:02,667-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [53659830] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:30:02,670-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [53659830] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53659830, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:30:02,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35014710 >2018-05-29 14:30:02,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:30:02,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:30:02,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:30:02,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35014710 >2018-05-29 14:30:02,773-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:30:02,786-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1ea27a5d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:30:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1ea27a5d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20f69683 >2018-05-29 14:30:02,789-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1ea27a5d] FINISH, SetVdsStatusVDSCommand, log id: 20f69683 >2018-05-29 14:30:02,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ea27a5d, Job ID: 759a1207-f58b-4f91-b42c-e15df233a4f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:30:02,825-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29903996, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:30:02,827-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1ea27a5d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29903996, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:30:02,838-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5f7193e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:30:02,839-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5f7193e6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6c1958d6] Autorecovering 2 hosts >2018-05-29 14:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6c1958d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [680d47fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f3c145e >2018-05-29 14:35:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [680d47fb] FINISH, SetVdsStatusVDSCommand, log id: 6f3c145e >2018-05-29 14:35:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [680d47fb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:35:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [680d47fb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ccee491 >2018-05-29 14:35:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] FINISH, SetVdsStatusVDSCommand, log id: 3ccee491 >2018-05-29 14:35:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 159ba4da >2018-05-29 14:35:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [73fa1e20] FINISH, SetHaMaintenanceModeVDSCommand, log id: 159ba4da >2018-05-29 14:35:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:35:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [73fa1e20] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:35:00,404-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5f08077] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e8d707a >2018-05-29 14:35:00,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [5f08077] FINISH, GetHardwareInfoVDSCommand, log id: 3e8d707a >2018-05-29 14:35:00,419-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6f6facd8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:35:00,427-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4bdc3416] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:35:00,432-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [222e6b94] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:35:00,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [222e6b94] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85849c4 >2018-05-29 14:35:00,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:35:00,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [222e6b94] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85849c4 >2018-05-29 14:35:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:00,817-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:00,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:00,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:00,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [222e6b94] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:00,829-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [222e6b94] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:35:00,832-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [222e6b94] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 222e6b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:35:00,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d2c66c >2018-05-29 14:35:00,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:35:00,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:35:00,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:35:00,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d2c66c >2018-05-29 14:35:00,935-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:35:00,948-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [44a6ad0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:35:00,949-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [44a6ad0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78dde6d4 >2018-05-29 14:35:00,951-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [44a6ad0] FINISH, SetVdsStatusVDSCommand, log id: 78dde6d4 >2018-05-29 14:35:00,982-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44a6ad0, Job ID: 4c94387b-3bca-46bb-9f93-98926d983a31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:35:00,987-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4bdc3416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:35:00,989-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [44a6ad0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4bdc3416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:35:01,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2f2a002b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:35:01,001-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2f2a002b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:35:01,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [34cc619b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 414b897d >2018-05-29 14:35:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [34cc619b] FINISH, GetHardwareInfoVDSCommand, log id: 414b897d >2018-05-29 14:35:01,705-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [265f44f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:35:01,714-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3fd7a2f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:35:01,720-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41e0ad92] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:35:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41e0ad92] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5cfb89d0 >2018-05-29 14:35:02,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:35:02,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41e0ad92] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5cfb89d0 >2018-05-29 14:35:02,114-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:02,115-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:02,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:02,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:02,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:35:02,124-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41e0ad92] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:35:02,128-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41e0ad92] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:35:02,130-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41e0ad92] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41e0ad92, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:35:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51f016a0 >2018-05-29 14:35:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:35:02,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:35:02,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:35:02,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51f016a0 >2018-05-29 14:35:02,230-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:35:02,244-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1445b4c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:35:02,245-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1445b4c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3562a9d7 >2018-05-29 14:35:02,247-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1445b4c1] FINISH, SetVdsStatusVDSCommand, log id: 3562a9d7 >2018-05-29 14:35:02,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1445b4c1, Job ID: 0d468c1c-a721-4bac-bf96-f7b3a705a18c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:35:02,281-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fd7a2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:35:02,283-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1445b4c1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fd7a2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:35:02,294-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [2fd078a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:35:02,295-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2fd078a9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [30d191c8] Autorecovering 2 hosts >2018-05-29 14:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [30d191c8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:40:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6196e1d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 430131ce >2018-05-29 14:40:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6196e1d2] FINISH, SetVdsStatusVDSCommand, log id: 430131ce >2018-05-29 14:40:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6196e1d2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:40:00,050-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6196e1d2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a6fa248 >2018-05-29 14:40:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] FINISH, SetVdsStatusVDSCommand, log id: 3a6fa248 >2018-05-29 14:40:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3edaf3d6 >2018-05-29 14:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5c53c53f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3edaf3d6 >2018-05-29 14:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5c53c53f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:40:01,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3291650a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d123488 >2018-05-29 14:40:01,829-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3291650a] FINISH, GetHardwareInfoVDSCommand, log id: 4d123488 >2018-05-29 14:40:01,843-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [65f8e85c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:40:01,852-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [73ca19d3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:40:01,858-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [30113b7e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:40:01,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [30113b7e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 114c2b66 >2018-05-29 14:40:02,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:40:02,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [30113b7e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 114c2b66 >2018-05-29 14:40:02,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:02,252-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:02,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:02,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:02,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:02,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [30113b7e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:02,264-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [30113b7e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:40:02,267-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [30113b7e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30113b7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:40:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d461d52 >2018-05-29 14:40:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:40:02,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:40:02,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:40:02,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d461d52 >2018-05-29 14:40:02,374-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:40:02,388-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2956e6e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:40:02,389-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2956e6e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a97b1ca >2018-05-29 14:40:02,391-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2956e6e3] FINISH, SetVdsStatusVDSCommand, log id: 6a97b1ca >2018-05-29 14:40:02,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2956e6e3, Job ID: 4bda97b5-476e-4c0b-9b17-f77eb3ebe388, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:40:02,404-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73ca19d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:40:02,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2956e6e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73ca19d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:40:02,418-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [51bec90] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:40:02,419-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [51bec90] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:40:03,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2b1b6052] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 27ae1b0f >2018-05-29 14:40:03,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2b1b6052] FINISH, GetHardwareInfoVDSCommand, log id: 27ae1b0f >2018-05-29 14:40:03,083-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [ebd033c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:40:03,088-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1c206c31] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:40:03,093-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1d99a9e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:40:03,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1d99a9e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b60f0ec >2018-05-29 14:40:05,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:40:05,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1d99a9e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b60f0ec >2018-05-29 14:40:05,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:05,429-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:05,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:05,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:05,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:40:05,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1d99a9e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:40:05,442-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1d99a9e4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:40:05,445-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d99a9e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1d99a9e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:40:05,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1206e0f4 >2018-05-29 14:40:05,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:40:05,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:40:05,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:40:05,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:40:05,543-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:40:05,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1206e0f4 >2018-05-29 14:40:05,544-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:40:05,557-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [b65b53c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:40:05,558-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b65b53c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ed887a8 >2018-05-29 14:40:05,560-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b65b53c] FINISH, SetVdsStatusVDSCommand, log id: 2ed887a8 >2018-05-29 14:40:05,566-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b65b53c, Job ID: 5587a1e7-0f85-493f-aa42-e623491df52a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:40:05,571-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c206c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:40:05,574-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b65b53c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c206c31, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:40:05,584-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [edc81c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:40:05,585-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [edc81c1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b75f368] Autorecovering 2 hosts >2018-05-29 14:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b75f368] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:45:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1680a402] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79d343b2 >2018-05-29 14:45:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1680a402] FINISH, SetVdsStatusVDSCommand, log id: 79d343b2 >2018-05-29 14:45:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1680a402] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:45:00,102-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1680a402] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:45:00,108-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:45:00,114-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:45:00,114-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,114-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,116-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ed8c0c7 >2018-05-29 14:45:00,118-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] FINISH, SetVdsStatusVDSCommand, log id: ed8c0c7 >2018-05-29 14:45:00,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 53452f5b >2018-05-29 14:45:00,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3b3e0edb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 53452f5b >2018-05-29 14:45:00,122-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:45:00,126-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3b3e0edb] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 3b3e0edb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 14:45:00,127-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3b3e0edb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:45:03,966-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [fb1d2ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: ab0fa49 >2018-05-29 14:45:03,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [fb1d2ab] FINISH, GetHardwareInfoVDSCommand, log id: ab0fa49 >2018-05-29 14:45:03,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7bc0206b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 23cd0ca >2018-05-29 14:45:03,978-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7bc0206b] FINISH, GetHardwareInfoVDSCommand, log id: 23cd0ca >2018-05-29 14:45:03,982-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [648d8eba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:45:03,991-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [ae7ec1f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:45:03,991-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [75fa2d04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:45:03,996-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [8b77312] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:45:04,000-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2effacc3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:45:04,005-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [8d8f58b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:45:04,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [8b77312] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3133e56b >2018-05-29 14:45:04,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [8d8f58b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 393222ee >2018-05-29 14:45:04,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:45:04,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [8d8f58b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 393222ee >2018-05-29 14:45:04,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:45:04,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [8b77312] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3133e56b >2018-05-29 14:45:04,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,400-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [8d8f58b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,406-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [8d8f58b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:45:04,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:45:04,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [8b77312] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:45:04,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8d8f58b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8d8f58b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:45:04,413-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [8b77312] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:45:04,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2697a9ff >2018-05-29 14:45:04,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:45:04,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [8b77312] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8b77312, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:45:04,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:45:04,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:45:04,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2697a9ff >2018-05-29 14:45:04,511-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:45:04,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28334ec8 >2018-05-29 14:45:04,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:45:04,525-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [47527ea4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:45:04,526-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [47527ea4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777a9af5 >2018-05-29 14:45:04,529-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [47527ea4] FINISH, SetVdsStatusVDSCommand, log id: 777a9af5 >2018-05-29 14:45:04,559-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 47527ea4, Job ID: 14f2f45c-27f0-46e4-bdf9-5110192830ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:45:04,564-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2effacc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:45:04,566-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [47527ea4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2effacc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:45:04,577-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7922e4c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:45:04,578-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7922e4c9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:45:04,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:45:04,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:45:04,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28334ec8 >2018-05-29 14:45:04,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:45:04,627-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [24875b2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:45:04,628-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [24875b2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cf1b8e0 >2018-05-29 14:45:04,630-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [24875b2e] FINISH, SetVdsStatusVDSCommand, log id: 4cf1b8e0 >2018-05-29 14:45:04,635-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24875b2e, Job ID: c56eabf1-2e51-42a5-aec3-efc285d34ffc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:45:04,640-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ae7ec1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:45:04,642-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [24875b2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ae7ec1f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:45:04,652-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [55b2719f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:45:04,653-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [55b2719f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [654d546c] Autorecovering 2 hosts >2018-05-29 14:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [654d546c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78c4993e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4244ff10 >2018-05-29 14:50:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78c4993e] FINISH, SetVdsStatusVDSCommand, log id: 4244ff10 >2018-05-29 14:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78c4993e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:50:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [78c4993e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:50:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b89edab >2018-05-29 14:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] FINISH, SetVdsStatusVDSCommand, log id: 3b89edab >2018-05-29 14:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3cf5b618 >2018-05-29 14:50:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [629bb0f8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3cf5b618 >2018-05-29 14:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:50:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [629bb0f8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:50:02,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4c8a2d47] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 216f9c25 >2018-05-29 14:50:02,238-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4c8a2d47] FINISH, GetHardwareInfoVDSCommand, log id: 216f9c25 >2018-05-29 14:50:02,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1d5eaa60] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64c7b637 >2018-05-29 14:50:02,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1d5eaa60] FINISH, GetHardwareInfoVDSCommand, log id: 64c7b637 >2018-05-29 14:50:02,253-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7fd90600] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:50:02,263-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [60c4d016] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:50:02,265-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4a14ea84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:50:02,268-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [23d8290f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:50:02,270-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [43852e18] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:50:02,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [23d8290f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3784ffd4 >2018-05-29 14:50:02,276-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [417668fa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:50:02,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [417668fa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 382b8909 >2018-05-29 14:50:02,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:50:02,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [23d8290f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3784ffd4 >2018-05-29 14:50:02,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:50:02,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [417668fa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 382b8909 >2018-05-29 14:50:02,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,667-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,668-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [23d8290f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:50:02,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [417668fa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:50:02,678-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [23d8290f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:50:02,681-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [417668fa] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:50:02,681-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [23d8290f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23d8290f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:50:02,688-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [417668fa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 417668fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:50:02,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b88a77a >2018-05-29 14:50:02,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:50:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:50:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:50:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b88a77a >2018-05-29 14:50:02,785-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:50:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 734f85f1 >2018-05-29 14:50:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:50:02,800-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [16c2eb38] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:50:02,801-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [16c2eb38] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 733d3244 >2018-05-29 14:50:02,803-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [16c2eb38] FINISH, SetVdsStatusVDSCommand, log id: 733d3244 >2018-05-29 14:50:02,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16c2eb38, Job ID: 09037de6-2c64-4494-bbe7-e22dbc869e2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:50:02,840-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60c4d016, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:50:02,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [16c2eb38] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60c4d016, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:50:02,853-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [71ea243b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:50:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [71ea243b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:50:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:50:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:50:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 734f85f1 >2018-05-29 14:50:02,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:50:02,899-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2ede377c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:50:02,900-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2ede377c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1deaa36a >2018-05-29 14:50:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2ede377c] FINISH, SetVdsStatusVDSCommand, log id: 1deaa36a >2018-05-29 14:50:02,908-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ede377c, Job ID: 39459e77-4b94-415a-adad-93d6e75988aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:50:02,913-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43852e18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:50:02,915-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2ede377c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43852e18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:50:02,926-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5fa486d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:50:02,927-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5fa486d9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:54:43,043-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [2cb20409] Attempting to update VMs/Templates Ovf. >2018-05-29 14:54:43,046-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 14:54:43,046-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 14:54:43,047-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:54:43,106-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 14:54:43,109-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 14:54:43,109-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 14:54:43,110-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 14:54:43,110-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 14:54:43,110-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 14:54:43,111-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [7229c8db] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 14:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71396ce1] Autorecovering 2 hosts >2018-05-29 14:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71396ce1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:55:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6ea5bae3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4165789c >2018-05-29 14:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6ea5bae3] FINISH, SetVdsStatusVDSCommand, log id: 4165789c >2018-05-29 14:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6ea5bae3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 14:55:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6ea5bae3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 14:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d3b8579 >2018-05-29 14:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] FINISH, SetVdsStatusVDSCommand, log id: 5d3b8579 >2018-05-29 14:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 645bc25 >2018-05-29 14:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [eef3d9c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 645bc25 >2018-05-29 14:55:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 14:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [eef3d9c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 14:55:00,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [269cb8e0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 29721de9 >2018-05-29 14:55:00,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [118c0464] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ec45c6e >2018-05-29 14:55:00,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [269cb8e0] FINISH, GetHardwareInfoVDSCommand, log id: 29721de9 >2018-05-29 14:55:00,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [118c0464] FINISH, GetHardwareInfoVDSCommand, log id: 6ec45c6e >2018-05-29 14:55:00,537-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [43aa17df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:55:00,540-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [71671723] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:55:00,546-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [505f10d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:55:00,549-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5709e504] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:55:00,551-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7ef89b88] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:55:00,554-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2946f922] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 14:55:00,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7ef89b88] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 693bcb36 >2018-05-29 14:55:00,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2946f922] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 307384c1 >2018-05-29 14:55:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:55:00,946-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7ef89b88] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 693bcb36 >2018-05-29 14:55:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:00,954-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:00,955-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:00,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:00,960-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7ef89b88] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:00,962-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7ef89b88] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:55:00,965-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ef89b88] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ef89b88, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:55:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55120e80 >2018-05-29 14:55:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:55:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:55:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:55:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55120e80 >2018-05-29 14:55:01,070-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:55:01,082-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1c1f8fd9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:55:01,083-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1c1f8fd9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47f37f5b >2018-05-29 14:55:01,085-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1c1f8fd9] FINISH, SetVdsStatusVDSCommand, log id: 47f37f5b >2018-05-29 14:55:01,117-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c1f8fd9, Job ID: 1f7ec829-49a4-418d-a0da-83b66eb36891, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:55:01,121-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 505f10d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:55:01,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1c1f8fd9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 505f10d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:55:01,134-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3dc3f7b6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 14:55:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3dc3f7b6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 14:55:01,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 14:55:01,932-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2946f922] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 307384c1 >2018-05-29 14:55:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:01,937-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:01,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:01,942-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:01,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 14:55:01,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2946f922] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 14:55:01,950-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2946f922] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 14:55:01,953-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2946f922] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2946f922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 14:55:01,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59080772 >2018-05-29 14:55:01,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:55:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 14:55:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 14:55:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59080772 >2018-05-29 14:55:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 14:55:02,070-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7dc5b050] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:55:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7dc5b050] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5724f032 >2018-05-29 14:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7dc5b050] FINISH, SetVdsStatusVDSCommand, log id: 5724f032 >2018-05-29 14:55:02,078-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7dc5b050, Job ID: be4c2e87-0852-460f-a03a-2b28f68ad6a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 14:55:02,083-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5709e504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 14:55:02,085-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7dc5b050] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5709e504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 14:55:02,095-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [39293cf6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 14:55:02,096-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [39293cf6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3575d8de] Autorecovering 2 hosts >2018-05-29 15:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3575d8de] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [729b10f8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cb39fcd >2018-05-29 15:00:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [729b10f8] FINISH, SetVdsStatusVDSCommand, log id: 4cb39fcd >2018-05-29 15:00:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,063-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [729b10f8] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 729b10f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 15:00:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [729b10f8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:00:00,064-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [729b10f8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:00:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4def288b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43404a77 >2018-05-29 15:00:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4def288b] FINISH, SetVdsStatusVDSCommand, log id: 43404a77 >2018-05-29 15:00:00,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4def288b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4fd72b39 >2018-05-29 15:00:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4def288b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4fd72b39 >2018-05-29 15:00:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:00:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4def288b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:00:01,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ad8cc7c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 12dde6c4 >2018-05-29 15:00:01,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6ad8cc7c] FINISH, GetHardwareInfoVDSCommand, log id: 12dde6c4 >2018-05-29 15:00:01,792-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [48e8a1d9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:00:01,800-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [6f5c6bd4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:00:01,805-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [60c8441c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:00:01,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [60c8441c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e1b6425 >2018-05-29 15:00:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:00:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [60c8441c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e1b6425 >2018-05-29 15:00:02,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:02,201-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:02,206-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:02,210-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:02,210-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [60c8441c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:02,214-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [60c8441c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:00:02,217-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60c8441c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 60c8441c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:00:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bf39bf0 >2018-05-29 15:00:02,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:00:02,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:00:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:00:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:00:02,316-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:00:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bf39bf0 >2018-05-29 15:00:02,317-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:00:02,330-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [406ee23] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:00:02,331-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [406ee23] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a1404e >2018-05-29 15:00:02,333-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [406ee23] FINISH, SetVdsStatusVDSCommand, log id: 3a1404e >2018-05-29 15:00:02,365-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 406ee23, Job ID: 59ca2fa5-5540-4b02-84cd-1ba8dd0643e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:00:02,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f5c6bd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:00:02,371-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [406ee23] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f5c6bd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:00:02,382-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [34ddb122] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:00:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [34ddb122] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:00:02,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [8151837] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64dcd54f >2018-05-29 15:00:02,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [8151837] FINISH, GetHardwareInfoVDSCommand, log id: 64dcd54f >2018-05-29 15:00:02,766-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6acf473e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:00:02,774-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [69b739c8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:00:02,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [694be675] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:00:02,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [694be675] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ea55e5d >2018-05-29 15:00:03,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:00:03,156-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [694be675] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ea55e5d >2018-05-29 15:00:03,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:03,160-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:03,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:03,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:00:03,170-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [694be675] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:00:03,172-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [694be675] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:00:03,175-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [694be675] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 694be675, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:00:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 63a532d9 >2018-05-29 15:00:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:00:03,277-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:00:03,277-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:00:03,277-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 63a532d9 >2018-05-29 15:00:03,277-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:00:03,290-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [43c94b41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:00:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [43c94b41] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5313dae8 >2018-05-29 15:00:03,293-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [43c94b41] FINISH, SetVdsStatusVDSCommand, log id: 5313dae8 >2018-05-29 15:00:03,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43c94b41, Job ID: f1a1811b-057f-4de0-b47d-4c0d9fc2ecbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:00:03,303-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 69b739c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:00:03,305-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [43c94b41] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 69b739c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:00:03,315-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3ae845f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:00:03,316-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3ae845f0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [14107945] Autorecovering 2 hosts >2018-05-29 15:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [14107945] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:05:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d7ec314] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 275e13d4 >2018-05-29 15:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d7ec314] FINISH, SetVdsStatusVDSCommand, log id: 275e13d4 >2018-05-29 15:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d7ec314] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:05:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d7ec314] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:05:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:05:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:05:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17c20070] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52e69fea >2018-05-29 15:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17c20070] FINISH, SetVdsStatusVDSCommand, log id: 52e69fea >2018-05-29 15:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17c20070] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7acbf263 >2018-05-29 15:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17c20070] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7acbf263 >2018-05-29 15:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:05:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17c20070] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:05:00,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [287b1a54] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4fadc395 >2018-05-29 15:05:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [287b1a54] FINISH, GetHardwareInfoVDSCommand, log id: 4fadc395 >2018-05-29 15:05:00,985-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [538da7d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:05:00,990-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1d8496a8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:05:00,995-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5414f21b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:05:01,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5414f21b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ac37d51 >2018-05-29 15:05:02,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:05:02,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5414f21b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ac37d51 >2018-05-29 15:05:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:02,384-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:02,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:02,388-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:02,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:02,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5414f21b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:02,439-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5414f21b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:05:02,443-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5414f21b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5414f21b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:05:02,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37f49a61 >2018-05-29 15:05:02,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:05:02,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37f49a61 >2018-05-29 15:05:02,548-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:05:02,562-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2d3ee45d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:05:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d3ee45d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b68f3a3 >2018-05-29 15:05:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d3ee45d] FINISH, SetVdsStatusVDSCommand, log id: 5b68f3a3 >2018-05-29 15:05:02,597-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2d3ee45d, Job ID: e8cb9ec0-6385-45c5-b429-1d4ad90324df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:05:02,602-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d8496a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:05:02,604-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2d3ee45d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d8496a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:05:02,615-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [68179374] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:05:02,616-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [68179374] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:05:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2fc9e613] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 55a73ded >2018-05-29 15:05:02,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [2fc9e613] FINISH, GetHardwareInfoVDSCommand, log id: 55a73ded >2018-05-29 15:05:03,011-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [74a64b0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:05:03,019-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4620e0da] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:05:03,024-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5461dac3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:05:03,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5461dac3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22f1c110 >2018-05-29 15:05:03,421-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:05:03,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5461dac3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22f1c110 >2018-05-29 15:05:03,424-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:03,425-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:03,430-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:03,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:05:03,434-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5461dac3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:05:03,437-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5461dac3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:05:03,439-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5461dac3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5461dac3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:05:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b3b84df >2018-05-29 15:05:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:05:03,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:05:03,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:05:03,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b3b84df >2018-05-29 15:05:03,542-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:05:03,555-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6919528] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:05:03,556-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6919528] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 550afee9 >2018-05-29 15:05:03,558-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6919528] FINISH, SetVdsStatusVDSCommand, log id: 550afee9 >2018-05-29 15:05:03,564-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6919528, Job ID: 036df527-fae7-4428-99bf-6e1c7cde5206, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:05:03,568-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4620e0da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:05:03,570-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6919528] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4620e0da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:05:03,580-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [109ef1a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:05:03,582-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [109ef1a8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [32ff4741] Autorecovering 2 hosts >2018-05-29 15:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [32ff4741] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:10:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6209675] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c9a8fd7 >2018-05-29 15:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6209675] FINISH, SetVdsStatusVDSCommand, log id: 2c9a8fd7 >2018-05-29 15:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [6209675] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:10:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [6209675] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:10:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b97b145 >2018-05-29 15:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] FINISH, SetVdsStatusVDSCommand, log id: 4b97b145 >2018-05-29 15:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52532b1d >2018-05-29 15:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3f6f4e29] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52532b1d >2018-05-29 15:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3f6f4e29] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:10:00,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [575a8689] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7f85a9c3 >2018-05-29 15:10:00,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [575a8689] FINISH, GetHardwareInfoVDSCommand, log id: 7f85a9c3 >2018-05-29 15:10:00,276-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [506a27d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:10:00,284-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [64a45fc5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:10:00,290-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [14554780] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:10:00,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [14554780] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fe2baa2 >2018-05-29 15:10:00,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:10:00,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [14554780] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fe2baa2 >2018-05-29 15:10:00,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:00,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:00,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:00,682-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:00,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:00,687-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [14554780] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:00,690-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [14554780] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:10:00,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [14554780] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 14554780, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:10:00,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cf7a6ac >2018-05-29 15:10:00,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:10:00,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:10:00,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:10:00,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cf7a6ac >2018-05-29 15:10:00,802-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:10:00,817-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5daeeebd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:10:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5daeeebd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 576669d1 >2018-05-29 15:10:00,820-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5daeeebd] FINISH, SetVdsStatusVDSCommand, log id: 576669d1 >2018-05-29 15:10:00,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5daeeebd, Job ID: 317d7c43-7c90-4398-bdcf-0728aef211da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:10:00,860-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64a45fc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:10:00,862-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5daeeebd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64a45fc5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:10:00,874-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [33e18823] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:10:00,875-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [33e18823] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:10:01,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [203ec591] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 763443f2 >2018-05-29 15:10:01,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [203ec591] FINISH, GetHardwareInfoVDSCommand, log id: 763443f2 >2018-05-29 15:10:01,295-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [66b3e822] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:10:01,303-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [55599a69] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:10:01,309-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [77e0e876] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:10:01,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [77e0e876] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 223c4c50 >2018-05-29 15:10:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:10:01,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [77e0e876] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 223c4c50 >2018-05-29 15:10:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:01,711-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:10:01,721-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [77e0e876] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:10:01,723-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [77e0e876] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:10:01,726-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [77e0e876] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77e0e876, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:10:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e0747b5 >2018-05-29 15:10:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:10:01,829-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:10:01,829-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:10:01,829-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e0747b5 >2018-05-29 15:10:01,829-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:10:01,843-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3d01fb4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:10:01,844-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d01fb4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c90bc80 >2018-05-29 15:10:01,846-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d01fb4d] FINISH, SetVdsStatusVDSCommand, log id: 6c90bc80 >2018-05-29 15:10:01,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d01fb4d, Job ID: 49af8d0b-b7e8-40a0-9346-0419f9d69226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:10:01,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55599a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:10:01,885-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d01fb4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55599a69, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:10:01,896-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1bd30273] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:10:01,897-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1bd30273] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [74531bf5] Autorecovering 2 hosts >2018-05-29 15:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [74531bf5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:15:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:15:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:15:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ac46f83] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52e74c40 >2018-05-29 15:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ac46f83] FINISH, SetVdsStatusVDSCommand, log id: 52e74c40 >2018-05-29 15:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ac46f83] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:15:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ac46f83] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:15:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f7aa801 >2018-05-29 15:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] FINISH, SetVdsStatusVDSCommand, log id: 1f7aa801 >2018-05-29 15:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 35798a1b >2018-05-29 15:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5b869bf6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 35798a1b >2018-05-29 15:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5b869bf6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:15:01,498-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [560e71c4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8b24c29 >2018-05-29 15:15:01,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [560e71c4] FINISH, GetHardwareInfoVDSCommand, log id: 8b24c29 >2018-05-29 15:15:01,513-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [c12d76a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:15:01,521-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [345273c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:15:01,526-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [396728c3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:15:01,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [396728c3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1d016147 >2018-05-29 15:15:01,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:15:01,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [396728c3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1d016147 >2018-05-29 15:15:01,910-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:01,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:01,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:01,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:01,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:01,920-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [396728c3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:01,923-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [396728c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:15:01,925-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [396728c3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 396728c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:15:01,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 182b0fe >2018-05-29 15:15:01,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:15:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:15:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:15:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 182b0fe >2018-05-29 15:15:02,030-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:15:02,045-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4fd1c8f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:15:02,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fd1c8f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 606e1597 >2018-05-29 15:15:02,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fd1c8f2] FINISH, SetVdsStatusVDSCommand, log id: 606e1597 >2018-05-29 15:15:02,084-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fd1c8f2, Job ID: 265af355-c98f-442b-b5aa-4a4bccaa302c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:15:02,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 345273c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:15:02,091-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4fd1c8f2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 345273c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:15:02,104-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3487ac01] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:15:02,105-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3487ac01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:15:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [221d6cae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78af985f >2018-05-29 15:15:02,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [221d6cae] FINISH, GetHardwareInfoVDSCommand, log id: 78af985f >2018-05-29 15:15:02,597-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [622c99b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:15:02,602-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [c44d32] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:15:02,607-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5bf160e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:15:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5bf160e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 473a8619 >2018-05-29 15:15:02,993-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:15:02,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5bf160e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 473a8619 >2018-05-29 15:15:02,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:02,998-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:03,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:03,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:03,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:15:03,008-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5bf160e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:15:03,010-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5bf160e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:15:03,013-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bf160e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5bf160e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:15:03,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 568fe6c7 >2018-05-29 15:15:03,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:15:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:15:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:15:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 568fe6c7 >2018-05-29 15:15:03,112-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:15:03,126-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [308d94bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:15:03,127-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [308d94bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3e30f8cf >2018-05-29 15:15:03,129-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [308d94bb] FINISH, SetVdsStatusVDSCommand, log id: 3e30f8cf >2018-05-29 15:15:03,158-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 308d94bb, Job ID: 5131c316-1f23-4db5-9ba8-176654dd60f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:15:03,163-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c44d32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:15:03,165-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [308d94bb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c44d32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:15:03,176-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [b48b976] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:15:03,177-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [b48b976] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [33587eef] Autorecovering 2 hosts >2018-05-29 15:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [33587eef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:20:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4e7f765c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53025302 >2018-05-29 15:20:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4e7f765c] FINISH, SetVdsStatusVDSCommand, log id: 53025302 >2018-05-29 15:20:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4e7f765c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:20:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [4e7f765c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:20:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6bb97728 >2018-05-29 15:20:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] FINISH, SetVdsStatusVDSCommand, log id: 6bb97728 >2018-05-29 15:20:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 153c4644 >2018-05-29 15:20:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [31c3f99d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 153c4644 >2018-05-29 15:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [31c3f99d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:20:00,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7ec2e7bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e61c37 >2018-05-29 15:20:00,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7ec2e7bb] FINISH, GetHardwareInfoVDSCommand, log id: 7e61c37 >2018-05-29 15:20:00,794-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [47b7b88a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:20:00,799-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5f0572bc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:20:00,804-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7fbe17ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:20:00,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7fbe17ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b79e3e5 >2018-05-29 15:20:01,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:20:01,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7fbe17ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b79e3e5 >2018-05-29 15:20:01,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:01,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:01,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:01,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:01,253-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:01,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7fbe17ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:01,256-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7fbe17ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:20:01,259-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fbe17ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fbe17ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:20:01,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f344710 >2018-05-29 15:20:01,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:20:01,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:20:01,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:20:01,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f344710 >2018-05-29 15:20:01,361-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:20:01,374-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [73da2e13] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:20:01,375-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [73da2e13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66042d49 >2018-05-29 15:20:01,377-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [73da2e13] FINISH, SetVdsStatusVDSCommand, log id: 66042d49 >2018-05-29 15:20:01,386-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73da2e13, Job ID: 293ed708-ce5a-4dca-8382-3e1c813d4eae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:20:01,390-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f0572bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:20:01,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [73da2e13] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f0572bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:20:01,404-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4a9d899c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:20:01,405-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4a9d899c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:20:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [55429984] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6d9d7c0f >2018-05-29 15:20:02,896-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [55429984] FINISH, GetHardwareInfoVDSCommand, log id: 6d9d7c0f >2018-05-29 15:20:02,937-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [247bba3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:20:02,942-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1ae11507] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:20:02,948-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e70bb72] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:20:02,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e70bb72] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 389fe0c3 >2018-05-29 15:20:03,337-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:20:03,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e70bb72] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 389fe0c3 >2018-05-29 15:20:03,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:03,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:03,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:03,346-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:03,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:20:03,351-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e70bb72] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:20:03,353-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e70bb72] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:20:03,356-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e70bb72] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e70bb72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:20:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 217b7c59 >2018-05-29 15:20:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:20:03,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:20:03,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:20:03,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 217b7c59 >2018-05-29 15:20:03,463-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:20:03,477-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [630c8905] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:20:03,478-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [630c8905] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e9ce2c1 >2018-05-29 15:20:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [630c8905] FINISH, SetVdsStatusVDSCommand, log id: 6e9ce2c1 >2018-05-29 15:20:03,515-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 630c8905, Job ID: 17754910-29c1-4cdd-910b-8c6a1b32752a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:20:03,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ae11507, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:20:03,522-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [630c8905] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ae11507, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:20:03,532-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3048f7cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:20:03,533-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3048f7cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:25:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b8b59eb] Autorecovering 2 hosts >2018-05-29 15:25:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b8b59eb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:25:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:25:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:25:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d659161] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 699bc4f2 >2018-05-29 15:25:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5d659161] FINISH, SetVdsStatusVDSCommand, log id: 699bc4f2 >2018-05-29 15:25:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5d659161] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:25:00,059-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5d659161] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:25:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225b29f9 >2018-05-29 15:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] FINISH, SetVdsStatusVDSCommand, log id: 225b29f9 >2018-05-29 15:25:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7a12e1d >2018-05-29 15:25:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6b3f2975] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7a12e1d >2018-05-29 15:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:25:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6b3f2975] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:25:01,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [16d3ab9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 58e87b1e >2018-05-29 15:25:01,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [16d3ab9] FINISH, GetHardwareInfoVDSCommand, log id: 58e87b1e >2018-05-29 15:25:01,290-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5bd9180d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:25:01,299-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4f31a2c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:25:01,304-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [58966f89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:25:01,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [58966f89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ccdbe64 >2018-05-29 15:25:02,258-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [535f3c90] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11f55a8f >2018-05-29 15:25:02,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [535f3c90] FINISH, GetHardwareInfoVDSCommand, log id: 11f55a8f >2018-05-29 15:25:02,280-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [10e9107e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:25:02,289-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [638b8bb1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:25:02,294-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2f986542] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:25:02,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2f986542] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12ca8ba9 >2018-05-29 15:25:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:25:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [58966f89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ccdbe64 >2018-05-29 15:25:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,681-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:25:02,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2f986542] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12ca8ba9 >2018-05-29 15:25:02,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,686-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [58966f89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,691-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [58966f89] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:25:02,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,694-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58966f89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58966f89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:25:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:25:02,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2f986542] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:25:02,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2796478e >2018-05-29 15:25:02,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:25:02,700-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2f986542] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:25:02,702-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f986542] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f986542, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:25:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:25:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:25:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2796478e >2018-05-29 15:25:02,792-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:25:02,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 629b08e0 >2018-05-29 15:25:02,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:25:02,806-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5069bfd8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:25:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5069bfd8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32f1addc >2018-05-29 15:25:02,809-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5069bfd8] FINISH, SetVdsStatusVDSCommand, log id: 32f1addc >2018-05-29 15:25:02,838-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5069bfd8, Job ID: 0de22425-82fa-49a9-b3dd-2301a8ddf6c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:25:02,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f31a2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:25:02,845-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5069bfd8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f31a2c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:25:02,856-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [66b54643] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:25:02,857-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [66b54643] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:25:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:25:02,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:25:02,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:25:02,899-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:25:02,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 629b08e0 >2018-05-29 15:25:02,899-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:25:02,912-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [40d80186] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:25:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [40d80186] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 574c3a88 >2018-05-29 15:25:02,915-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [40d80186] FINISH, SetVdsStatusVDSCommand, log id: 574c3a88 >2018-05-29 15:25:02,921-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40d80186, Job ID: 0f3ba775-d7e1-43a8-863e-c305431b9eab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:25:02,925-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 638b8bb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:25:02,927-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [40d80186] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 638b8bb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:25:02,938-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [545f9812] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:25:02,939-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [545f9812] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4f5ce181] Autorecovering 2 hosts >2018-05-29 15:30:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4f5ce181] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:30:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [292c4ecf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fc20e2 >2018-05-29 15:30:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [292c4ecf] FINISH, SetVdsStatusVDSCommand, log id: fc20e2 >2018-05-29 15:30:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [292c4ecf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:30:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [292c4ecf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:30:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6c681466] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49a79bb3 >2018-05-29 15:30:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6c681466] FINISH, SetVdsStatusVDSCommand, log id: 49a79bb3 >2018-05-29 15:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6c681466] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 342a1c6 >2018-05-29 15:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6c681466] FINISH, SetHaMaintenanceModeVDSCommand, log id: 342a1c6 >2018-05-29 15:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6c681466] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:30:00,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [66bbe354] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 46274c6b >2018-05-29 15:30:00,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [66bbe354] FINISH, GetHardwareInfoVDSCommand, log id: 46274c6b >2018-05-29 15:30:00,514-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [21c9828a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:30:00,523-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [125e3d17] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:30:00,528-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [646f0236] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:30:00,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [646f0236] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53895a2e >2018-05-29 15:30:00,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [294ff49] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7650b311 >2018-05-29 15:30:00,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [294ff49] FINISH, GetHardwareInfoVDSCommand, log id: 7650b311 >2018-05-29 15:30:00,607-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [23925a75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:30:00,616-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [77dad3fc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:30:00,621-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [23924d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:30:00,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [23924d5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c4d58e1 >2018-05-29 15:30:00,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:30:00,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [646f0236] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53895a2e >2018-05-29 15:30:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:00,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:00,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:00,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [646f0236] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:00,987-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [646f0236] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:30:00,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [646f0236] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 646f0236, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:30:00,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a35924b >2018-05-29 15:30:00,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:30:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:30:01,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [23924d5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2c4d58e1 >2018-05-29 15:30:01,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:01,009-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:01,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:01,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:30:01,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [23924d5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:30:01,072-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [23924d5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:30:01,076-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [23924d5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23924d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:30:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:30:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:30:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a35924b >2018-05-29 15:30:01,095-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:30:01,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6babc3b6 >2018-05-29 15:30:01,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:30:01,111-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7095ad8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:30:01,112-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7095ad8a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 585123b5 >2018-05-29 15:30:01,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7095ad8a] FINISH, SetVdsStatusVDSCommand, log id: 585123b5 >2018-05-29 15:30:01,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7095ad8a, Job ID: c7ed451d-29d3-48c5-8381-3d46de42a0b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:30:01,128-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 125e3d17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:30:01,130-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7095ad8a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 125e3d17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:30:01,141-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [59599534] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:30:01,142-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [59599534] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:30:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:30:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:30:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6babc3b6 >2018-05-29 15:30:01,200-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:30:01,213-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7c5e42de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:30:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7c5e42de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 173f208a >2018-05-29 15:30:01,216-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7c5e42de] FINISH, SetVdsStatusVDSCommand, log id: 173f208a >2018-05-29 15:30:01,222-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c5e42de, Job ID: c7f83e4b-f271-4106-b31d-41e7b42ad124, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:30:01,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77dad3fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:30:01,229-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7c5e42de] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77dad3fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:30:01,239-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [502f731a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:30:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [502f731a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7328dbdf] Autorecovering 2 hosts >2018-05-29 15:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7328dbdf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [542685e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65eb0110 >2018-05-29 15:35:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [542685e] FINISH, SetVdsStatusVDSCommand, log id: 65eb0110 >2018-05-29 15:35:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [542685e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:35:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [542685e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cf7bc29 >2018-05-29 15:35:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] FINISH, SetVdsStatusVDSCommand, log id: 7cf7bc29 >2018-05-29 15:35:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 43ceb93d >2018-05-29 15:35:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [562f6d4f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 43ceb93d >2018-05-29 15:35:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:35:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [562f6d4f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:35:01,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3da14bea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 192891c5 >2018-05-29 15:35:01,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3da14bea] FINISH, GetHardwareInfoVDSCommand, log id: 192891c5 >2018-05-29 15:35:01,821-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [436a3845] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:35:01,830-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5bba2c14] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:35:01,836-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [bfea202] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:35:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [bfea202] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 261d3fdf >2018-05-29 15:35:01,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2032a8fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 13955862 >2018-05-29 15:35:01,966-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2032a8fc] FINISH, GetHardwareInfoVDSCommand, log id: 13955862 >2018-05-29 15:35:01,980-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [52f6cc78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:35:01,984-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [652929ba] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:35:01,989-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21a99b50] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:35:01,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21a99b50] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: e581bc3 >2018-05-29 15:35:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:35:02,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [bfea202] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 261d3fdf >2018-05-29 15:35:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:02,235-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:02,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:02,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:02,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [bfea202] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:02,243-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [bfea202] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:35:02,245-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [bfea202] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bfea202, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:35:02,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7597470f >2018-05-29 15:35:02,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:35:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:35:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:35:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7597470f >2018-05-29 15:35:02,346-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:35:02,360-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4d5f0ae4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:35:02,361-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d5f0ae4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64d5aced >2018-05-29 15:35:02,363-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4d5f0ae4] FINISH, SetVdsStatusVDSCommand, log id: 64d5aced >2018-05-29 15:35:02,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d5f0ae4, Job ID: 917e04a1-d919-4e9a-b076-6179a867d971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:35:02,400-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bba2c14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:35:02,402-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4d5f0ae4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bba2c14, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:35:02,414-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6d087502] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:35:02,415-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6d087502] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:35:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:35:03,369-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [21a99b50] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: e581bc3 >2018-05-29 15:35:03,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:03,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:03,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:03,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:35:03,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [21a99b50] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:35:03,387-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [21a99b50] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:35:03,390-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [21a99b50] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21a99b50, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:35:03,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5dc56b7c >2018-05-29 15:35:03,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:35:03,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:35:03,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:35:03,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5dc56b7c >2018-05-29 15:35:03,492-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:35:03,506-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [10176208] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:35:03,507-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10176208] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5699b4e0 >2018-05-29 15:35:03,509-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10176208] FINISH, SetVdsStatusVDSCommand, log id: 5699b4e0 >2018-05-29 15:35:03,542-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 10176208, Job ID: a7489940-b382-453a-8e8e-e86e3b2b15a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:35:03,547-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 652929ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:35:03,549-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [10176208] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 652929ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:35:03,560-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [4fcf05b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:35:03,561-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4fcf05b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45d652ae] Autorecovering 2 hosts >2018-05-29 15:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45d652ae] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:40:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:40:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:40:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [393cab7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d80c >2018-05-29 15:40:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [393cab7] FINISH, SetVdsStatusVDSCommand, log id: 1d80c >2018-05-29 15:40:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [393cab7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:40:00,059-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [393cab7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 657bec >2018-05-29 15:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] FINISH, SetVdsStatusVDSCommand, log id: 657bec >2018-05-29 15:40:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 9b66ec4 >2018-05-29 15:40:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6a0907ca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 9b66ec4 >2018-05-29 15:40:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:40:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6a0907ca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:40:01,317-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5140f9c2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2de495e9 >2018-05-29 15:40:01,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5140f9c2] FINISH, GetHardwareInfoVDSCommand, log id: 2de495e9 >2018-05-29 15:40:01,333-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [51e46d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:40:01,341-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [227bfab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:40:01,347-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68ac57e1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:40:01,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68ac57e1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: eeccae4 >2018-05-29 15:40:01,720-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:40:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [68ac57e1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: eeccae4 >2018-05-29 15:40:01,724-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:01,725-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:01,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:01,729-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:01,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:01,734-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [68ac57e1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:01,737-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [68ac57e1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:40:01,739-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [68ac57e1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68ac57e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:40:01,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37736488 >2018-05-29 15:40:01,743-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:40:01,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:40:01,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:40:01,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37736488 >2018-05-29 15:40:01,843-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:40:01,856-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2b5d1815] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:40:01,858-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b5d1815] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fff12a7 >2018-05-29 15:40:01,860-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b5d1815] FINISH, SetVdsStatusVDSCommand, log id: 6fff12a7 >2018-05-29 15:40:01,889-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b5d1815, Job ID: 21f55bb3-34ad-48f3-b999-717bbb438840, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:40:01,893-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 227bfab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:40:01,895-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b5d1815] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 227bfab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:40:01,906-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3d0be4b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:40:01,907-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3d0be4b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:40:03,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [51499c04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 152d9a1 >2018-05-29 15:40:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [51499c04] FINISH, GetHardwareInfoVDSCommand, log id: 152d9a1 >2018-05-29 15:40:03,024-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3d9c6237] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:40:03,032-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2273bffd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:40:03,037-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [52ddac85] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:40:03,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [52ddac85] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 295c3e67 >2018-05-29 15:40:03,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:40:03,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [52ddac85] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 295c3e67 >2018-05-29 15:40:03,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:03,435-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:03,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:03,439-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:03,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:40:03,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [52ddac85] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:40:03,446-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [52ddac85] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:40:03,449-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [52ddac85] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ddac85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:40:03,455-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9a10141 >2018-05-29 15:40:03,455-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:40:03,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:40:03,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:40:03,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9a10141 >2018-05-29 15:40:03,554-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:40:03,567-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [45df9312] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:40:03,569-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45df9312] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b39f18 >2018-05-29 15:40:03,571-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [45df9312] FINISH, SetVdsStatusVDSCommand, log id: 43b39f18 >2018-05-29 15:40:03,603-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 45df9312, Job ID: 6d29a7bc-bb36-41e0-8213-af8b98e29212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:40:03,608-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2273bffd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:40:03,610-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [45df9312] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2273bffd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:40:03,621-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [67474c72] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:40:03,622-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [67474c72] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [645b6db7] Autorecovering 2 hosts >2018-05-29 15:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [645b6db7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:45:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [434a458a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d947d2d >2018-05-29 15:45:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [434a458a] FINISH, SetVdsStatusVDSCommand, log id: 3d947d2d >2018-05-29 15:45:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [434a458a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:45:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [434a458a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:45:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:45:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:45:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [560220c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40bbf578 >2018-05-29 15:45:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [560220c3] FINISH, SetVdsStatusVDSCommand, log id: 40bbf578 >2018-05-29 15:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [560220c3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e9a97e8 >2018-05-29 15:45:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [560220c3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e9a97e8 >2018-05-29 15:45:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:45:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [560220c3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:45:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49723e73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 726c922f >2018-05-29 15:45:01,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49723e73] FINISH, GetHardwareInfoVDSCommand, log id: 726c922f >2018-05-29 15:45:01,239-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [328ecb60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:45:01,247-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [dc7d39e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:45:01,253-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [49e0f815] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:45:01,264-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [49e0f815] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6467ea21 >2018-05-29 15:45:01,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:45:01,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [49e0f815] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6467ea21 >2018-05-29 15:45:01,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:01,650-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:01,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:01,654-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:01,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:01,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [49e0f815] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:01,662-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [49e0f815] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:45:01,664-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [49e0f815] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49e0f815, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:45:01,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b7b775 >2018-05-29 15:45:01,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:45:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:45:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:45:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b7b775 >2018-05-29 15:45:01,768-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:45:01,781-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6278c01d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:45:01,782-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6278c01d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2619c3a7 >2018-05-29 15:45:01,785-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6278c01d] FINISH, SetVdsStatusVDSCommand, log id: 2619c3a7 >2018-05-29 15:45:01,816-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6278c01d, Job ID: d5cbe33f-ed85-4aad-8889-9e60d8fd65f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:45:01,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: dc7d39e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:45:01,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6278c01d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: dc7d39e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:45:01,834-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [52db3ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:45:01,835-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [52db3ed7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:45:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3eb6194f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 37122590 >2018-05-29 15:45:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3eb6194f] FINISH, GetHardwareInfoVDSCommand, log id: 37122590 >2018-05-29 15:45:03,416-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [26a881e9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:45:03,420-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2e96624] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:45:03,425-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5d9aa8c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:45:03,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5d9aa8c8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: df606ae >2018-05-29 15:45:04,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:45:04,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5d9aa8c8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: df606ae >2018-05-29 15:45:04,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:04,820-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:04,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:04,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:04,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:45:04,829-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5d9aa8c8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:45:04,832-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5d9aa8c8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:45:04,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d9aa8c8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d9aa8c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:45:04,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26489997 >2018-05-29 15:45:04,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:45:04,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:45:04,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:45:04,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:45:04,941-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:45:04,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26489997 >2018-05-29 15:45:04,941-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:45:04,954-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3080b85d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:45:04,955-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3080b85d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42691712 >2018-05-29 15:45:04,958-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3080b85d] FINISH, SetVdsStatusVDSCommand, log id: 42691712 >2018-05-29 15:45:04,966-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3080b85d, Job ID: b70f5b50-29ae-430e-96d3-04c6636f26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:45:04,970-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e96624, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:45:04,972-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3080b85d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e96624, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:45:04,983-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5d6802b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:45:04,984-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5d6802b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [67c6b3b6] Autorecovering 2 hosts >2018-05-29 15:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [67c6b3b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [28b04aa8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e5c8f9b >2018-05-29 15:50:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [28b04aa8] FINISH, SetVdsStatusVDSCommand, log id: 4e5c8f9b >2018-05-29 15:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [28b04aa8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:50:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [28b04aa8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53492d19] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 114c958f >2018-05-29 15:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [53492d19] FINISH, SetVdsStatusVDSCommand, log id: 114c958f >2018-05-29 15:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [53492d19] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7aafd7c0 >2018-05-29 15:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [53492d19] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7aafd7c0 >2018-05-29 15:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [53492d19] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:50:02,514-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3dde8b2f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2b698320 >2018-05-29 15:50:02,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3dde8b2f] FINISH, GetHardwareInfoVDSCommand, log id: 2b698320 >2018-05-29 15:50:02,532-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [1224cb10] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:50:02,540-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [10dbee0b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:50:02,545-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [c30c7ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:50:02,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [c30c7ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f7b84a4 >2018-05-29 15:50:02,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [33980152] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 69eca898 >2018-05-29 15:50:02,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [33980152] FINISH, GetHardwareInfoVDSCommand, log id: 69eca898 >2018-05-29 15:50:02,652-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [74159147] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:50:02,656-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [60a1530d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:50:02,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1628c7e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:50:02,667-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1628c7e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24ea45b6 >2018-05-29 15:50:02,923-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:50:02,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [c30c7ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f7b84a4 >2018-05-29 15:50:02,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:02,928-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:02,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:02,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:02,937-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [c30c7ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:02,940-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [c30c7ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:50:02,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [c30c7ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c30c7ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:50:02,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67ef1256 >2018-05-29 15:50:02,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:50:03,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:50:03,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1628c7e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24ea45b6 >2018-05-29 15:50:03,039-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:50:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:50:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:50:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67ef1256 >2018-05-29 15:50:03,040-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:50:03,044-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:03,045-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:03,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:50:03,049-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1628c7e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:50:03,053-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1628c7e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:50:03,054-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4a15802a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:50:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a15802a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b398117 >2018-05-29 15:50:03,056-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1628c7e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1628c7e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:50:03,058-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4a15802a] FINISH, SetVdsStatusVDSCommand, log id: 1b398117 >2018-05-29 15:50:03,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 691ee5f9 >2018-05-29 15:50:03,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:50:03,090-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a15802a, Job ID: 554a89cf-2774-4566-a643-4608acb59057, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:50:03,094-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10dbee0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:50:03,096-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4a15802a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10dbee0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:50:03,107-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [33b40f53] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:50:03,108-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [33b40f53] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:50:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:50:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:50:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 691ee5f9 >2018-05-29 15:50:03,167-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:50:03,180-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7d7b8a71] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:50:03,182-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7d7b8a71] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ec6f657 >2018-05-29 15:50:03,184-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7d7b8a71] FINISH, SetVdsStatusVDSCommand, log id: 2ec6f657 >2018-05-29 15:50:03,189-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7d7b8a71, Job ID: fd608107-724a-481d-97c0-395ce46bd619, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:50:03,194-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60a1530d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:50:03,196-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d7b8a71] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60a1530d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:50:03,207-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [42c5a076] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:50:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [42c5a076] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:54:43,111-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [36e40224] Attempting to update VMs/Templates Ovf. >2018-05-29 15:54:43,113-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 15:54:43,113-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 15:54:43,114-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:54:43,118-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 15:54:43,122-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 15:54:43,122-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 15:54:43,122-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 15:54:43,122-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 15:54:43,123-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 15:54:43,123-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [5401fcd8] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 15:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7654db52] Autorecovering 2 hosts >2018-05-29 15:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7654db52] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:55:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c0edefc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4892dc8c >2018-05-29 15:55:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c0edefc] FINISH, SetVdsStatusVDSCommand, log id: 4892dc8c >2018-05-29 15:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c0edefc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 15:55:00,070-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5c0edefc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 15:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79cc579 >2018-05-29 15:55:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] FINISH, SetVdsStatusVDSCommand, log id: 79cc579 >2018-05-29 15:55:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a85be57 >2018-05-29 15:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2e0a378c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a85be57 >2018-05-29 15:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 15:55:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2e0a378c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 15:55:00,774-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7d6d0714] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7609b05d >2018-05-29 15:55:00,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7d6d0714] FINISH, GetHardwareInfoVDSCommand, log id: 7609b05d >2018-05-29 15:55:00,789-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [52101437] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:55:00,798-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [46e25abe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:55:00,804-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [74462c64] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:55:00,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [74462c64] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3cf08468 >2018-05-29 15:55:00,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3341c7a0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 679c4fe8 >2018-05-29 15:55:00,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3341c7a0] FINISH, GetHardwareInfoVDSCommand, log id: 679c4fe8 >2018-05-29 15:55:00,837-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3c15e46d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:55:00,842-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6190ab55] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:55:00,847-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4f4c45f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 15:55:00,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4f4c45f2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a73d93 >2018-05-29 15:55:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:55:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [74462c64] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3cf08468 >2018-05-29 15:55:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [74462c64] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,203-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [74462c64] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:55:01,205-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [74462c64] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74462c64, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:55:01,209-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: df31f54 >2018-05-29 15:55:01,209-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:55:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 15:55:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [4f4c45f2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a73d93 >2018-05-29 15:55:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,237-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 15:55:01,238-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [4f4c45f2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 15:55:01,242-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [4f4c45f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 15:55:01,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4f4c45f2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f4c45f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:55:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:55:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:55:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: df31f54 >2018-05-29 15:55:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:55:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 545af1c9 >2018-05-29 15:55:01,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 15:55:01,322-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [131f1ef1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:55:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [131f1ef1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146bb75f >2018-05-29 15:55:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [131f1ef1] FINISH, SetVdsStatusVDSCommand, log id: 146bb75f >2018-05-29 15:55:01,331-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 131f1ef1, Job ID: bf282c6c-0b02-428b-877d-ed51d05182cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:55:01,336-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 46e25abe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:55:01,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [131f1ef1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 46e25abe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:55:01,349-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [4244e574] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 15:55:01,350-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4244e574] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:55:01,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 15:55:01,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 15:55:01,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 545af1c9 >2018-05-29 15:55:01,411-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 15:55:01,426-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [44abec7b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:55:01,427-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [44abec7b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: adb7688 >2018-05-29 15:55:01,429-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [44abec7b] FINISH, SetVdsStatusVDSCommand, log id: adb7688 >2018-05-29 15:55:01,435-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44abec7b, Job ID: c4ad25b7-5326-4c3a-a2f2-b0e3032423c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 15:55:01,440-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6190ab55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 15:55:01,442-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44abec7b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6190ab55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 15:55:01,454-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5c05533] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 15:55:01,455-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5c05533] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f41ebf0] Autorecovering 2 hosts >2018-05-29 16:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f41ebf0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:00:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [547d9d05] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ecf3ef2 >2018-05-29 16:00:02,000-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [547d9d05] FINISH, SetVdsStatusVDSCommand, log id: 1ecf3ef2 >2018-05-29 16:00:02,002-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:02,004-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [547d9d05] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:00:02,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [547d9d05] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:00:02,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:00:02,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:00:02,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:02,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:02,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [496ee15c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aee52e2 >2018-05-29 16:00:02,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [496ee15c] FINISH, SetVdsStatusVDSCommand, log id: 5aee52e2 >2018-05-29 16:00:02,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [496ee15c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6dbc773f >2018-05-29 16:00:02,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [496ee15c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6dbc773f >2018-05-29 16:00:02,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:00:02,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [496ee15c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:00:04,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5b1cf189] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 348be048 >2018-05-29 16:00:04,989-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5b1cf189] FINISH, GetHardwareInfoVDSCommand, log id: 348be048 >2018-05-29 16:00:05,004-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [717bada0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:00:05,013-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [52978606] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:00:05,018-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4ab5ceae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:00:05,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4ab5ceae] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 174d5502 >2018-05-29 16:00:05,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [28a6ae81] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: ca381cb >2018-05-29 16:00:05,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [28a6ae81] FINISH, GetHardwareInfoVDSCommand, log id: ca381cb >2018-05-29 16:00:05,265-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [c02b904] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:00:05,274-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [38164f3d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:00:05,279-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3f004761] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:00:05,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3f004761] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ba3144d >2018-05-29 16:00:05,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:00:05,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4ab5ceae] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 174d5502 >2018-05-29 16:00:05,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,463-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,472-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4ab5ceae] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,475-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4ab5ceae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:00:05,477-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4ab5ceae] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4ab5ceae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:00:05,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52589623 >2018-05-29 16:00:05,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:00:05,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:00:05,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:00:05,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:00:05,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:00:05,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52589623 >2018-05-29 16:00:05,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:00:05,595-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2b2a2d84] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:00:05,597-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2b2a2d84] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 743f0516 >2018-05-29 16:00:05,599-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2b2a2d84] FINISH, SetVdsStatusVDSCommand, log id: 743f0516 >2018-05-29 16:00:05,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b2a2d84, Job ID: 33c9828d-4973-4b25-9e53-5ed70ea71970, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:00:05,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52978606, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:00:05,638-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2b2a2d84] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52978606, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:00:05,650-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [9622801] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:00:05,651-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [9622801] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:00:05,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:00:05,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3f004761] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ba3144d >2018-05-29 16:00:05,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,664-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:00:05,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3f004761] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:00:05,672-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3f004761] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:00:05,674-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3f004761] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3f004761, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:00:05,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17859ada >2018-05-29 16:00:05,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:00:05,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:00:05,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:00:05,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17859ada >2018-05-29 16:00:05,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:00:05,789-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [66ef70a7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:00:05,790-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66ef70a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f4cd28 >2018-05-29 16:00:05,792-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66ef70a7] FINISH, SetVdsStatusVDSCommand, log id: f4cd28 >2018-05-29 16:00:05,798-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 66ef70a7, Job ID: 9dd3db22-6c6a-4706-be66-6173ea6af62e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:00:05,802-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 38164f3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:00:05,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [66ef70a7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 38164f3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:00:05,815-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6998dd5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:00:05,816-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6998dd5d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4adff942] Autorecovering 2 hosts >2018-05-29 16:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4adff942] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:05:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:05:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:05:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [23ce6d26] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 219e3dd7 >2018-05-29 16:05:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [23ce6d26] FINISH, SetVdsStatusVDSCommand, log id: 219e3dd7 >2018-05-29 16:05:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [23ce6d26] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:05:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [23ce6d26] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:05:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [51554fc6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2dcf9d64 >2018-05-29 16:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [51554fc6] FINISH, SetVdsStatusVDSCommand, log id: 2dcf9d64 >2018-05-29 16:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [51554fc6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 673465b5 >2018-05-29 16:05:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [51554fc6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 673465b5 >2018-05-29 16:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [51554fc6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:05:00,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [14fd3aee] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 577fc57d >2018-05-29 16:05:00,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [14fd3aee] FINISH, GetHardwareInfoVDSCommand, log id: 577fc57d >2018-05-29 16:05:00,353-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3b8362f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:05:00,361-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7e052f85] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:05:00,366-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5160606a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:05:00,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5160606a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 20634e83 >2018-05-29 16:05:00,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75092fd1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6583c1ce >2018-05-29 16:05:00,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75092fd1] FINISH, GetHardwareInfoVDSCommand, log id: 6583c1ce >2018-05-29 16:05:00,439-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [e81b9b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:05:00,443-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3188594e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:05:00,448-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e6942dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:05:00,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e6942dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22bcc966 >2018-05-29 16:05:00,762-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:05:00,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5160606a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 20634e83 >2018-05-29 16:05:00,766-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,767-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,771-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,776-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5160606a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5160606a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:05:00,782-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5160606a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5160606a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:05:00,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 183757a2 >2018-05-29 16:05:00,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:05:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:05:00,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e6942dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22bcc966 >2018-05-29 16:05:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,834-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,838-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,839-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:05:00,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e6942dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:05:00,846-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e6942dc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:05:00,849-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e6942dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e6942dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:05:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:05:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:05:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 183757a2 >2018-05-29 16:05:00,884-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:05:00,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215f29f >2018-05-29 16:05:00,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:05:00,897-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [608804f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:05:00,898-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [608804f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2977c589 >2018-05-29 16:05:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [608804f6] FINISH, SetVdsStatusVDSCommand, log id: 2977c589 >2018-05-29 16:05:00,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 608804f6, Job ID: 57d26a64-4d8a-437e-a69a-9a2489154ac8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:05:00,935-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e052f85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:05:00,937-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [608804f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e052f85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:05:00,947-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6233025a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:05:00,948-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6233025a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:05:00,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:05:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:05:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:05:00,986-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:05:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215f29f >2018-05-29 16:05:00,987-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:05:00,999-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3aa50b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:05:01,000-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3aa50b1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72e4bcac >2018-05-29 16:05:01,002-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3aa50b1c] FINISH, SetVdsStatusVDSCommand, log id: 72e4bcac >2018-05-29 16:05:01,008-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3aa50b1c, Job ID: 413fe6ee-5596-4a10-9c32-cff3be3b249f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:05:01,012-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3188594e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:05:01,014-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3aa50b1c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3188594e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:05:01,024-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [47dfa90f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:05:01,025-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [47dfa90f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [39a55cdc] Autorecovering 2 hosts >2018-05-29 16:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [39a55cdc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4eb008a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 605b523f >2018-05-29 16:10:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4eb008a3] FINISH, SetVdsStatusVDSCommand, log id: 605b523f >2018-05-29 16:10:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4eb008a3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:10:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4eb008a3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:10:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:10:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f65207b >2018-05-29 16:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] FINISH, SetVdsStatusVDSCommand, log id: 4f65207b >2018-05-29 16:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3f43b440 >2018-05-29 16:10:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3fd2fd00] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3f43b440 >2018-05-29 16:10:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:10:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3fd2fd00] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:10:01,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bc3ac7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6d262e76 >2018-05-29 16:10:01,545-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bc3ac7] FINISH, GetHardwareInfoVDSCommand, log id: 6d262e76 >2018-05-29 16:10:01,557-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [46487940] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:10:01,566-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [f1614e7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:10:01,571-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [680e8482] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:10:01,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [680e8482] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3cd0839 >2018-05-29 16:10:01,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5401d6ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 77d8aefe >2018-05-29 16:10:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5401d6ad] FINISH, GetHardwareInfoVDSCommand, log id: 77d8aefe >2018-05-29 16:10:01,703-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1e52f455] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:10:01,707-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7a4a70f6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:10:01,712-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [74211327] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:10:01,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [74211327] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 43d0fb39 >2018-05-29 16:10:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:10:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [680e8482] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3cd0839 >2018-05-29 16:10:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:01,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:01,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:01,971-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:01,976-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [680e8482] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:01,978-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [680e8482] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:10:01,981-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [680e8482] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 680e8482, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:10:01,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1586e3e0 >2018-05-29 16:10:01,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:10:02,084-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:10:02,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [74211327] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 43d0fb39 >2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:10:02,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:10:02,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:10:02,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1586e3e0 >2018-05-29 16:10:02,086-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:10:02,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:02,089-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:02,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:02,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:02,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:10:02,098-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [74211327] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:10:02,099-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [34dd554a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:10:02,101-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [34dd554a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 122acba5 >2018-05-29 16:10:02,102-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [74211327] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:10:02,103-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [34dd554a] FINISH, SetVdsStatusVDSCommand, log id: 122acba5 >2018-05-29 16:10:02,104-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [74211327] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74211327, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:10:02,111-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34dd554a, Job ID: 2282faba-cf02-4841-8e33-63609d981d2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:10:02,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53bc0fd1 >2018-05-29 16:10:02,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:10:02,116-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f1614e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:10:02,118-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [34dd554a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f1614e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:10:02,130-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [29183894] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:10:02,131-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [29183894] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:10:02,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:10:02,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:10:02,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53bc0fd1 >2018-05-29 16:10:02,208-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:10:02,221-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [25fa59bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:10:02,223-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [25fa59bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a9a6052 >2018-05-29 16:10:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [25fa59bb] FINISH, SetVdsStatusVDSCommand, log id: 5a9a6052 >2018-05-29 16:10:02,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 25fa59bb, Job ID: d5dbdd28-c2e0-44dc-a853-7424bbf3f278, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:10:02,259-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a4a70f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:10:02,261-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [25fa59bb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a4a70f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:10:02,272-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [548d2db9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:10:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [548d2db9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [491e366] Autorecovering 2 hosts >2018-05-29 16:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [491e366] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:15:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [17e8824f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c10554e >2018-05-29 16:15:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [17e8824f] FINISH, SetVdsStatusVDSCommand, log id: 2c10554e >2018-05-29 16:15:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [17e8824f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:15:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [17e8824f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2731b2de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e6757b3 >2018-05-29 16:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2731b2de] FINISH, SetVdsStatusVDSCommand, log id: 1e6757b3 >2018-05-29 16:15:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2731b2de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 71ab610b >2018-05-29 16:15:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2731b2de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 71ab610b >2018-05-29 16:15:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:15:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2731b2de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:15:02,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33b0d5d9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60c3c86 >2018-05-29 16:15:02,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [33b0d5d9] FINISH, GetHardwareInfoVDSCommand, log id: 60c3c86 >2018-05-29 16:15:02,832-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [733227b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:15:02,840-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [35388ecc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:15:02,846-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [671f9a16] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:15:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [671f9a16] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 103bbca7 >2018-05-29 16:15:02,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7e4443bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 49788350 >2018-05-29 16:15:02,932-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7e4443bf] FINISH, GetHardwareInfoVDSCommand, log id: 49788350 >2018-05-29 16:15:02,944-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [60d7c783] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:15:02,949-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [565a242] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:15:02,954-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4d80c05c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:15:02,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4d80c05c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15b9b2c5 >2018-05-29 16:15:03,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:15:03,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [671f9a16] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 103bbca7 >2018-05-29 16:15:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,253-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,254-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [671f9a16] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,256-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [671f9a16] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:15:03,259-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [671f9a16] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 671f9a16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:15:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19cb8f4b >2018-05-29 16:15:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:15:03,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:15:03,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4d80c05c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15b9b2c5 >2018-05-29 16:15:03,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,340-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,344-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:15:03,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4d80c05c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:15:03,352-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4d80c05c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:15:03,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4d80c05c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d80c05c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:15:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:15:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:15:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19cb8f4b >2018-05-29 16:15:03,363-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:15:03,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48386d8 >2018-05-29 16:15:03,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:15:03,377-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4fa0575c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:15:03,378-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4fa0575c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a511eb4 >2018-05-29 16:15:03,380-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4fa0575c] FINISH, SetVdsStatusVDSCommand, log id: 4a511eb4 >2018-05-29 16:15:03,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4fa0575c, Job ID: 74e39d50-21b9-434b-b50a-c51635d488b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:15:03,416-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35388ecc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:15:03,419-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4fa0575c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35388ecc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:15:03,430-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [73b073e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:15:03,431-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [73b073e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:15:03,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:15:03,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:15:03,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48386d8 >2018-05-29 16:15:03,468-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:15:03,482-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [124f47a0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:15:03,483-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [124f47a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46dab786 >2018-05-29 16:15:03,485-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [124f47a0] FINISH, SetVdsStatusVDSCommand, log id: 46dab786 >2018-05-29 16:15:03,491-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 124f47a0, Job ID: e4d72331-8c7a-45f8-89af-e27566a71c55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:15:03,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 565a242, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:15:03,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [124f47a0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 565a242, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:15:03,508-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [66169357] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:15:03,509-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [66169357] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [209b15a0] Autorecovering 2 hosts >2018-05-29 16:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [209b15a0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:20:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4ca8d7a2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 98ce42b >2018-05-29 16:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4ca8d7a2] FINISH, SetVdsStatusVDSCommand, log id: 98ce42b >2018-05-29 16:20:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4ca8d7a2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:20:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4ca8d7a2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:20:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a008bde] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67c8688e >2018-05-29 16:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4a008bde] FINISH, SetVdsStatusVDSCommand, log id: 67c8688e >2018-05-29 16:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4a008bde] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 574649b0 >2018-05-29 16:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4a008bde] FINISH, SetHaMaintenanceModeVDSCommand, log id: 574649b0 >2018-05-29 16:20:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:20:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4a008bde] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:20:00,992-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [14079fd8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 74448439 >2018-05-29 16:20:00,994-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [14079fd8] FINISH, GetHardwareInfoVDSCommand, log id: 74448439 >2018-05-29 16:20:01,007-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7041252f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:20:01,012-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [237ef9e2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:20:01,017-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [54ae9f62] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:20:01,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [54ae9f62] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4638a16d >2018-05-29 16:20:01,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [32688010] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 35b7f834 >2018-05-29 16:20:01,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [32688010] FINISH, GetHardwareInfoVDSCommand, log id: 35b7f834 >2018-05-29 16:20:01,178-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5e9d3da2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:20:01,182-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6635f5ac] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:20:01,187-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [376fe387] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:20:01,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [376fe387] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 297e6c7d >2018-05-29 16:20:01,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:20:01,398-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [54ae9f62] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4638a16d >2018-05-29 16:20:01,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:01,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:01,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:01,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:01,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:01,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [54ae9f62] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:01,415-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [54ae9f62] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:20:01,418-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [54ae9f62] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54ae9f62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:20:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2581ba0c >2018-05-29 16:20:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:20:01,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:20:01,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:20:01,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2581ba0c >2018-05-29 16:20:01,515-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:20:01,528-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7c7e12f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:20:01,530-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7c7e12f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f5e077e >2018-05-29 16:20:01,532-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7c7e12f] FINISH, SetVdsStatusVDSCommand, log id: f5e077e >2018-05-29 16:20:01,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c7e12f, Job ID: b174ac97-88cc-4b8f-9459-6c3ccc71c219, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:20:01,565-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 237ef9e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:20:01,567-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7c7e12f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 237ef9e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:20:01,578-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [8148f28] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:20:01,579-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [8148f28] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:20:02,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:20:02,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [376fe387] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 297e6c7d >2018-05-29 16:20:02,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:02,563-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:02,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:02,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:02,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:20:02,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [376fe387] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:20:02,575-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [376fe387] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:20:02,577-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [376fe387] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 376fe387, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:20:02,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 671bd7d0 >2018-05-29 16:20:02,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:20:02,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:20:02,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:20:02,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 671bd7d0 >2018-05-29 16:20:02,677-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:20:02,690-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2891b3f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:20:02,691-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2891b3f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 200b7280 >2018-05-29 16:20:02,693-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2891b3f1] FINISH, SetVdsStatusVDSCommand, log id: 200b7280 >2018-05-29 16:20:02,699-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2891b3f1, Job ID: a1711388-a525-4b76-9651-2ab50ef90ea1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:20:02,703-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6635f5ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:20:02,706-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2891b3f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6635f5ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:20:02,716-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [34f2f5ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:20:02,717-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [34f2f5ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [144179a1] Autorecovering 2 hosts >2018-05-29 16:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [144179a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:25:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d4a2aa5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5cb4cdb9 >2018-05-29 16:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d4a2aa5] FINISH, SetVdsStatusVDSCommand, log id: 5cb4cdb9 >2018-05-29 16:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3d4a2aa5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:25:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3d4a2aa5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:25:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:25:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [633aaf78] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3fe5fe88 >2018-05-29 16:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [633aaf78] FINISH, SetVdsStatusVDSCommand, log id: 3fe5fe88 >2018-05-29 16:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [633aaf78] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4156c074 >2018-05-29 16:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [633aaf78] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4156c074 >2018-05-29 16:25:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:25:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [633aaf78] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:25:00,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7d3de395] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 369157e0 >2018-05-29 16:25:00,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7d3de395] FINISH, GetHardwareInfoVDSCommand, log id: 369157e0 >2018-05-29 16:25:00,499-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6ffa23e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:25:00,507-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7a393305] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:25:00,512-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4d1e54d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:25:00,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4d1e54d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31807b94 >2018-05-29 16:25:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:25:00,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4d1e54d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31807b94 >2018-05-29 16:25:00,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:00,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:00,952-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:00,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:00,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:00,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4d1e54d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:00,960-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4d1e54d3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:25:00,963-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4d1e54d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d1e54d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:25:00,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 200e0ebb >2018-05-29 16:25:00,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:25:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:25:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:25:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 200e0ebb >2018-05-29 16:25:01,069-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:25:01,083-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [3fc66d0b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:25:01,084-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3fc66d0b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 633443cd >2018-05-29 16:25:01,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3fc66d0b] FINISH, SetVdsStatusVDSCommand, log id: 633443cd >2018-05-29 16:25:01,118-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3fc66d0b, Job ID: 2cbdbfcf-68dd-48d4-a8c7-1970f5ed7e78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:25:01,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7a393305, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:25:01,125-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3fc66d0b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7a393305, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:25:01,136-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [32a25855] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:25:01,137-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [32a25855] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:25:02,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [719827d4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8a0a309 >2018-05-29 16:25:02,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [719827d4] FINISH, GetHardwareInfoVDSCommand, log id: 8a0a309 >2018-05-29 16:25:02,203-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [162508b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:25:02,207-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [62cd7dc7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:25:02,212-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3aaaf4fd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:25:02,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3aaaf4fd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f7af0b5 >2018-05-29 16:25:02,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:25:02,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3aaaf4fd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f7af0b5 >2018-05-29 16:25:02,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:02,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:02,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:25:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3aaaf4fd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:25:02,617-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3aaaf4fd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:25:02,620-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3aaaf4fd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3aaaf4fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:25:02,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 56ca181c >2018-05-29 16:25:02,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:25:02,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:25:02,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:25:02,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:25:02,722-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:25:02,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 56ca181c >2018-05-29 16:25:02,722-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:25:02,735-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f0c0676] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:25:02,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f0c0676] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61f3ae3d >2018-05-29 16:25:02,738-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f0c0676] FINISH, SetVdsStatusVDSCommand, log id: 61f3ae3d >2018-05-29 16:25:02,744-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f0c0676, Job ID: 6dbeac57-789a-4022-ae9e-40440c904d56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:25:02,749-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62cd7dc7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:25:02,751-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f0c0676] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62cd7dc7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:25:02,762-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [34ccda9f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:25:02,763-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [34ccda9f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3ad168e] Autorecovering 2 hosts >2018-05-29 16:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3ad168e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:30:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:30:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:30:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7dfbb65e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f31bf6b >2018-05-29 16:30:00,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7dfbb65e] FINISH, SetVdsStatusVDSCommand, log id: 2f31bf6b >2018-05-29 16:30:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7dfbb65e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:30:00,057-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfbb65e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [19ebda78] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d4b6066 >2018-05-29 16:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [19ebda78] FINISH, SetVdsStatusVDSCommand, log id: 3d4b6066 >2018-05-29 16:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [19ebda78] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 60384b5a >2018-05-29 16:30:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [19ebda78] FINISH, SetHaMaintenanceModeVDSCommand, log id: 60384b5a >2018-05-29 16:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [19ebda78] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:30:00,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [589ef885] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 69efa4e6 >2018-05-29 16:30:00,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [589ef885] FINISH, GetHardwareInfoVDSCommand, log id: 69efa4e6 >2018-05-29 16:30:00,396-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [15a8a3f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:30:00,404-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6979b4cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:30:00,409-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c75920] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:30:00,416-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c75920] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a1ae9e9 >2018-05-29 16:30:00,795-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:30:00,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [c75920] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a1ae9e9 >2018-05-29 16:30:00,799-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:00,800-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:00,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:00,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:00,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:00,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [c75920] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:00,812-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [c75920] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:30:00,815-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [c75920] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c75920, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:30:00,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 679b7dda >2018-05-29 16:30:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:30:00,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:30:00,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:30:00,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 679b7dda >2018-05-29 16:30:00,916-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:30:00,929-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7ca078b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:30:00,930-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7ca078b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65cf1cbf >2018-05-29 16:30:00,932-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7ca078b0] FINISH, SetVdsStatusVDSCommand, log id: 65cf1cbf >2018-05-29 16:30:00,961-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7ca078b0, Job ID: f05fef4f-d2ed-47eb-91ec-270d3c06349a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:30:00,966-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6979b4cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:30:00,968-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7ca078b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6979b4cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:30:00,979-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [35c1897f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:30:00,980-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [35c1897f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:30:01,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [101168b4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 288ab8c8 >2018-05-29 16:30:01,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [101168b4] FINISH, GetHardwareInfoVDSCommand, log id: 288ab8c8 >2018-05-29 16:30:01,922-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [248946c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:30:01,926-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4df473b7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:30:01,932-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [365ea687] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:30:01,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [365ea687] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e1fcc09 >2018-05-29 16:30:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:30:03,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [365ea687] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e1fcc09 >2018-05-29 16:30:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:03,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:03,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:03,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:30:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [365ea687] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:30:03,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [365ea687] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:30:03,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [365ea687] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 365ea687, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:30:03,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29caa681 >2018-05-29 16:30:03,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:30:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:30:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:30:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29caa681 >2018-05-29 16:30:03,442-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:30:03,455-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6b5fa6a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:30:03,457-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b5fa6a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5101be0f >2018-05-29 16:30:03,459-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b5fa6a8] FINISH, SetVdsStatusVDSCommand, log id: 5101be0f >2018-05-29 16:30:03,491-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5fa6a8, Job ID: c85f99f0-5866-4cfe-bdde-e7c4b2e212df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:30:03,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4df473b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:30:03,497-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6b5fa6a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4df473b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:30:03,508-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [46c45a27] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:30:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [46c45a27] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3588bf55] Autorecovering 2 hosts >2018-05-29 16:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3588bf55] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:35:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f2a329] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1d8b7a59 >2018-05-29 16:35:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f2a329] FINISH, SetVdsStatusVDSCommand, log id: 1d8b7a59 >2018-05-29 16:35:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f2a329] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:35:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3f2a329] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:35:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:35:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:35:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5d55fc6 >2018-05-29 16:35:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] FINISH, SetVdsStatusVDSCommand, log id: 5d55fc6 >2018-05-29 16:35:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7aecab0f >2018-05-29 16:35:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2c2c1d5f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7aecab0f >2018-05-29 16:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2c2c1d5f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:35:01,165-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1ed9f88c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ab486cf >2018-05-29 16:35:01,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1ed9f88c] FINISH, GetHardwareInfoVDSCommand, log id: 6ab486cf >2018-05-29 16:35:01,179-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7523ecbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:35:01,184-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [66b06f36] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:35:01,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d8b4e8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:35:01,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d8b4e8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ce3133e >2018-05-29 16:35:01,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:35:01,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d8b4e8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ce3133e >2018-05-29 16:35:01,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:01,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:01,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:01,587-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:01,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6a7878cd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3c8b6725 >2018-05-29 16:35:01,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6a7878cd] FINISH, GetHardwareInfoVDSCommand, log id: 3c8b6725 >2018-05-29 16:35:01,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:01,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d8b4e8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:01,596-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d8b4e8f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:35:01,599-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d8b4e8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d8b4e8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:35:01,604-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [497765c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:35:01,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c7d4304 >2018-05-29 16:35:01,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:35:01,613-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [13f626a2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:35:01,619-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [720d011f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:35:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [720d011f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ec531a0 >2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:35:01,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:35:01,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:35:01,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c7d4304 >2018-05-29 16:35:01,709-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:35:01,722-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3e5d01cd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:35:01,723-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e5d01cd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57ef08ae >2018-05-29 16:35:01,725-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3e5d01cd] FINISH, SetVdsStatusVDSCommand, log id: 57ef08ae >2018-05-29 16:35:01,734-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3e5d01cd, Job ID: 4c3d3532-fd42-46a4-b5e2-96a4b0ab7576, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:35:01,739-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66b06f36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:35:01,741-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3e5d01cd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66b06f36, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:35:01,751-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [86a3c2a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:35:01,752-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [86a3c2a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:35:02,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:35:02,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [720d011f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ec531a0 >2018-05-29 16:35:02,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:02,013-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:02,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:02,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:35:02,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [720d011f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:35:02,025-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [720d011f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:35:02,027-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [720d011f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 720d011f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:35:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 673a6c4c >2018-05-29 16:35:02,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:35:02,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:35:02,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:35:02,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 673a6c4c >2018-05-29 16:35:02,123-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:35:02,136-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [368dfbd4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:35:02,137-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [368dfbd4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4228af8c >2018-05-29 16:35:02,139-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [368dfbd4] FINISH, SetVdsStatusVDSCommand, log id: 4228af8c >2018-05-29 16:35:02,144-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 368dfbd4, Job ID: ba888fd1-82a8-408c-beaa-ede2714572c9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:35:02,149-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 13f626a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:35:02,151-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [368dfbd4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 13f626a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:35:02,161-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2d1b579] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:35:02,162-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2d1b579] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1d313705] Autorecovering 2 hosts >2018-05-29 16:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1d313705] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:40:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4790f22a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a10ea8c >2018-05-29 16:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4790f22a] FINISH, SetVdsStatusVDSCommand, log id: 6a10ea8c >2018-05-29 16:40:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [4790f22a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:40:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [4790f22a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:40:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [798ab0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a7eacee >2018-05-29 16:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [798ab0] FINISH, SetVdsStatusVDSCommand, log id: a7eacee >2018-05-29 16:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [798ab0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 293c2234 >2018-05-29 16:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [798ab0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 293c2234 >2018-05-29 16:40:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:40:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [798ab0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:40:02,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ed88466] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f2283f4 >2018-05-29 16:40:02,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4ed88466] FINISH, GetHardwareInfoVDSCommand, log id: 6f2283f4 >2018-05-29 16:40:02,733-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f85e9e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:40:02,741-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3b202975] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:40:02,746-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [74268304] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:40:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [74268304] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ec9c7d5 >2018-05-29 16:40:02,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5019642c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 694636f1 >2018-05-29 16:40:02,821-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5019642c] FINISH, GetHardwareInfoVDSCommand, log id: 694636f1 >2018-05-29 16:40:02,834-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55067287] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:40:02,838-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1d3cde0c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:40:02,843-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [dfc4b96] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:40:02,849-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [dfc4b96] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72e7a2b6 >2018-05-29 16:40:03,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:40:03,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [74268304] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ec9c7d5 >2018-05-29 16:40:03,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [74268304] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,150-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [74268304] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:40:03,153-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [74268304] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74268304, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:40:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6e4a4322 >2018-05-29 16:40:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:40:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:40:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [dfc4b96] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72e7a2b6 >2018-05-29 16:40:03,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,238-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:40:03,239-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [dfc4b96] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:40:03,242-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [dfc4b96] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:40:03,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [dfc4b96] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: dfc4b96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:40:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:40:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:40:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6e4a4322 >2018-05-29 16:40:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:40:03,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 179273ab >2018-05-29 16:40:03,258-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:40:03,269-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [42261683] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:40:03,270-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42261683] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 777b9c42 >2018-05-29 16:40:03,272-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42261683] FINISH, SetVdsStatusVDSCommand, log id: 777b9c42 >2018-05-29 16:40:03,304-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42261683, Job ID: ae4db035-847c-40fe-ae09-72763b7452ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:40:03,308-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b202975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:40:03,310-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42261683] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b202975, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:40:03,321-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3c6ce366] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:40:03,322-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3c6ce366] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:40:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:40:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:40:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 179273ab >2018-05-29 16:40:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:40:03,373-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1902e309] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:40:03,374-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1902e309] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78f4c3bd >2018-05-29 16:40:03,376-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1902e309] FINISH, SetVdsStatusVDSCommand, log id: 78f4c3bd >2018-05-29 16:40:03,382-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1902e309, Job ID: 17277738-04c8-4421-899c-b4a19660aded, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:40:03,386-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d3cde0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:40:03,388-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1902e309] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d3cde0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:40:03,398-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1682215a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:40:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1682215a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3300d87b] Autorecovering 2 hosts >2018-05-29 16:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3300d87b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [237aeaa1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bde7ed8 >2018-05-29 16:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [237aeaa1] FINISH, SetVdsStatusVDSCommand, log id: 2bde7ed8 >2018-05-29 16:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [237aeaa1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [237aeaa1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [743c44d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a0df1be >2018-05-29 16:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [743c44d3] FINISH, SetVdsStatusVDSCommand, log id: 1a0df1be >2018-05-29 16:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [743c44d3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b172d66 >2018-05-29 16:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [743c44d3] FINISH, SetHaMaintenanceModeVDSCommand, log id: b172d66 >2018-05-29 16:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [743c44d3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:45:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ed26e96] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5bab4d4 >2018-05-29 16:45:00,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6ed26e96] FINISH, GetHardwareInfoVDSCommand, log id: 5bab4d4 >2018-05-29 16:45:01,001-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1affc51d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:45:01,010-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [70e30a63] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:45:01,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6bd9e4dd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44900058 >2018-05-29 16:45:01,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6bd9e4dd] FINISH, GetHardwareInfoVDSCommand, log id: 44900058 >2018-05-29 16:45:01,016-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7853cff6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:45:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7853cff6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6fb982f1 >2018-05-29 16:45:01,028-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2db36ee5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:45:01,036-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [74748c0c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:45:01,042-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c5e6361] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:45:01,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c5e6361] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 68aa231e >2018-05-29 16:45:01,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:45:01,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7853cff6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6fb982f1 >2018-05-29 16:45:01,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,401-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,405-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7853cff6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,413-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7853cff6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:45:01,416-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7853cff6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7853cff6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:45:01,422-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4526d006 >2018-05-29 16:45:01,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:45:01,431-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:45:01,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3c5e6361] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 68aa231e >2018-05-29 16:45:01,435-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,436-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,440-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:45:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3c5e6361] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:45:01,447-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3c5e6361] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:45:01,450-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3c5e6361] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c5e6361, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:45:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:45:01,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:45:01,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:45:01,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:45:01,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4526d006 >2018-05-29 16:45:01,522-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:45:01,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64345ac6 >2018-05-29 16:45:01,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:45:01,535-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4a16a74] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:45:01,536-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a16a74] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c85abb0 >2018-05-29 16:45:01,539-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4a16a74] FINISH, SetVdsStatusVDSCommand, log id: 4c85abb0 >2018-05-29 16:45:01,570-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a16a74, Job ID: d5af2478-562e-48bf-b5e9-f588775c0ce2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:45:01,574-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70e30a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:45:01,576-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4a16a74] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70e30a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:45:01,587-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5e4597ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:45:01,588-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5e4597ec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:45:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:45:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:45:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64345ac6 >2018-05-29 16:45:01,623-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:45:01,636-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [42928c56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:45:01,637-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [42928c56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7522148d >2018-05-29 16:45:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [42928c56] FINISH, SetVdsStatusVDSCommand, log id: 7522148d >2018-05-29 16:45:01,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42928c56, Job ID: 4491f178-d135-496a-b5fc-c358a3fed938, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:45:01,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74748c0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:45:01,650-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [42928c56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74748c0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:45:01,661-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7222eac1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:45:01,662-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7222eac1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:50:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1dd176a1] Autorecovering 2 hosts >2018-05-29 16:50:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1dd176a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:50:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29e28c60] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5fc0f2b5 >2018-05-29 16:50:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29e28c60] FINISH, SetVdsStatusVDSCommand, log id: 5fc0f2b5 >2018-05-29 16:50:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [29e28c60] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:50:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [29e28c60] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:50:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:50:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:50:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2254df9e >2018-05-29 16:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] FINISH, SetVdsStatusVDSCommand, log id: 2254df9e >2018-05-29 16:50:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 10a0306d >2018-05-29 16:50:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [35bcce4e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 10a0306d >2018-05-29 16:50:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:50:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [35bcce4e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:50:02,369-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6389d15d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 65385da4 >2018-05-29 16:50:02,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [6389d15d] FINISH, GetHardwareInfoVDSCommand, log id: 65385da4 >2018-05-29 16:50:02,384-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7d82e8a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:50:02,388-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1e973eb4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:50:02,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7aeaf68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5dbb17fa >2018-05-29 16:50:02,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7aeaf68] FINISH, GetHardwareInfoVDSCommand, log id: 5dbb17fa >2018-05-29 16:50:02,394-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5084914f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:50:02,407-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [643de52a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:50:02,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5084914f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7b65bcd4 >2018-05-29 16:50:02,411-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [509975ac] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:50:02,416-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [77264588] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:50:02,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [77264588] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78a7ee80 >2018-05-29 16:50:02,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:50:02,772-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5084914f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7b65bcd4 >2018-05-29 16:50:02,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5084914f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,789-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5084914f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:50:02,791-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5084914f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5084914f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:50:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 480d6cb8 >2018-05-29 16:50:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:50:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:50:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [77264588] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78a7ee80 >2018-05-29 16:50:02,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,803-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,807-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,807-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,811-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:50:02,812-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [77264588] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:50:02,815-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [77264588] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:50:02,818-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [77264588] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77264588, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:50:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:50:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:50:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 480d6cb8 >2018-05-29 16:50:02,895-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:50:02,901-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4868a953 >2018-05-29 16:50:02,901-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:50:02,909-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [20df8233] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:50:02,910-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [20df8233] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656cc772 >2018-05-29 16:50:02,912-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [20df8233] FINISH, SetVdsStatusVDSCommand, log id: 656cc772 >2018-05-29 16:50:02,921-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20df8233, Job ID: f1ceb27b-5436-40b7-a84e-57e23c4f5638, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:50:02,925-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e973eb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:50:02,928-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [20df8233] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e973eb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:50:02,939-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [38ba64d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:50:02,940-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [38ba64d9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:50:02,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4868a953 >2018-05-29 16:50:02,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:50:03,010-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4ec673b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:50:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4ec673b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 769fef2 >2018-05-29 16:50:03,013-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4ec673b0] FINISH, SetVdsStatusVDSCommand, log id: 769fef2 >2018-05-29 16:50:03,019-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4ec673b0, Job ID: 443d13ad-15b9-4ee1-82cd-d6d8d022105c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:50:03,023-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 509975ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:50:03,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ec673b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 509975ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:50:03,038-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3a681f65] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:50:03,039-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3a681f65] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:54:43,123-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [1d4b1893] Attempting to update VMs/Templates Ovf. >2018-05-29 16:54:43,126-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 16:54:43,126-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 16:54:43,126-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:54:43,131-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 16:54:43,135-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 16:54:43,135-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 16:54:43,135-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 16:54:43,135-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 16:54:43,136-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 16:54:43,136-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [6fa24a16] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 16:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [257852a1] Autorecovering 2 hosts >2018-05-29 16:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [257852a1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a343a87] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62b35175 >2018-05-29 16:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7a343a87] FINISH, SetVdsStatusVDSCommand, log id: 62b35175 >2018-05-29 16:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [7a343a87] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 16:55:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a343a87] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:55:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 16:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6178b23f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454ae6aa >2018-05-29 16:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6178b23f] FINISH, SetVdsStatusVDSCommand, log id: 454ae6aa >2018-05-29 16:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6178b23f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 205b20fa >2018-05-29 16:55:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6178b23f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 205b20fa >2018-05-29 16:55:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 16:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6178b23f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 16:55:00,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [664583b0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ccc6cff >2018-05-29 16:55:00,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2b5cc194] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 67d3824a >2018-05-29 16:55:00,724-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [664583b0] FINISH, GetHardwareInfoVDSCommand, log id: 6ccc6cff >2018-05-29 16:55:00,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2b5cc194] FINISH, GetHardwareInfoVDSCommand, log id: 67d3824a >2018-05-29 16:55:00,740-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [55419cfd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:55:00,742-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [527fcd88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:55:00,750-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [45326a77] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:55:00,751-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [51c13625] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:55:00,756-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30a9593c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:55:00,757-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [508b8f90] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 16:55:00,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30a9593c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 478591ce >2018-05-29 16:55:00,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [508b8f90] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 789b0442 >2018-05-29 16:55:01,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:55:01,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [30a9593c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 478591ce >2018-05-29 16:55:01,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,149-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 16:55:01,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [508b8f90] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 789b0442 >2018-05-29 16:55:01,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [30a9593c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,155-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,156-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [30a9593c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:55:01,159-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [30a9593c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30a9593c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:55:01,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4d408bd >2018-05-29 16:55:01,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:55:01,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 16:55:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [508b8f90] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 16:55:01,169-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [508b8f90] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 16:55:01,172-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [508b8f90] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 508b8f90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:55:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:55:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:55:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4d408bd >2018-05-29 16:55:01,261-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:55:01,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 766d15ab >2018-05-29 16:55:01,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 16:55:01,274-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4b0db259] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:55:01,275-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4b0db259] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: db515 >2018-05-29 16:55:01,277-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4b0db259] FINISH, SetVdsStatusVDSCommand, log id: db515 >2018-05-29 16:55:01,306-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b0db259, Job ID: 5ca51d00-4ebe-4a08-b15d-86735cd163aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:55:01,310-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45326a77, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:55:01,312-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b0db259] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45326a77, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:55:01,323-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2cd85270] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 16:55:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2cd85270] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:55:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 16:55:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 16:55:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 766d15ab >2018-05-29 16:55:01,362-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 16:55:01,374-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [6ebd8f2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:55:01,375-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6ebd8f2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75dff24 >2018-05-29 16:55:01,377-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [6ebd8f2c] FINISH, SetVdsStatusVDSCommand, log id: 75dff24 >2018-05-29 16:55:01,383-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6ebd8f2c, Job ID: dcd9e8df-7296-4ab9-b1b0-375027643a1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 16:55:01,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 51c13625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 16:55:01,389-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ebd8f2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 51c13625, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 16:55:01,400-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [1e9a34b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 16:55:01,401-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [1e9a34b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:00:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [67792366] Autorecovering 2 hosts >2018-05-29 17:00:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [67792366] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:00:00,013-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:00:00,020-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:00:00,020-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,020-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,022-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c924a64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48a84318 >2018-05-29 17:00:00,026-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c924a64] FINISH, SetVdsStatusVDSCommand, log id: 48a84318 >2018-05-29 17:00:00,027-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,028-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [c924a64] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:00:00,028-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [c924a64] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:00:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:00:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:00:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b9f77f7 >2018-05-29 17:00:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] FINISH, SetVdsStatusVDSCommand, log id: 6b9f77f7 >2018-05-29 17:00:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 58bc8c75 >2018-05-29 17:00:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [6d8669c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 58bc8c75 >2018-05-29 17:00:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:00:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6d8669c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:00:01,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [148e16ab] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1befa970 >2018-05-29 17:00:01,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [148e16ab] FINISH, GetHardwareInfoVDSCommand, log id: 1befa970 >2018-05-29 17:00:01,969-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2ab56c7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:00:01,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7933049e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78241f86 >2018-05-29 17:00:01,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7933049e] FINISH, GetHardwareInfoVDSCommand, log id: 78241f86 >2018-05-29 17:00:01,978-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [342355ba] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:00:01,983-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [33beda13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:00:01,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [33beda13] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a0a5ba3 >2018-05-29 17:00:01,995-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1783666c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:00:01,999-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2e2f6d9f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:00:02,005-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [410905b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:00:02,017-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [410905b9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31cdcf83 >2018-05-29 17:00:02,390-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:00:02,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [410905b9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31cdcf83 >2018-05-29 17:00:02,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:02,395-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:02,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:02,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:02,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:02,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [410905b9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:02,407-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [410905b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:00:02,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [410905b9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 410905b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:00:02,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 32e670d3 >2018-05-29 17:00:02,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:00:02,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:00:02,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:00:02,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 32e670d3 >2018-05-29 17:00:02,511-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:00:02,524-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [287116d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:00:02,525-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [287116d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c4b8f04 >2018-05-29 17:00:02,528-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [287116d8] FINISH, SetVdsStatusVDSCommand, log id: 6c4b8f04 >2018-05-29 17:00:02,561-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 287116d8, Job ID: 09283aa5-a2fe-4079-8a94-f62c85498139, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:00:02,565-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e2f6d9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:00:02,568-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [287116d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e2f6d9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:00:02,578-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [78abae48] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:00:02,580-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [78abae48] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:00:03,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:00:03,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [33beda13] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a0a5ba3 >2018-05-29 17:00:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:03,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:03,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:03,365-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:00:03,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [33beda13] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:00:03,373-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [33beda13] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:00:03,376-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [33beda13] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 33beda13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:00:03,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4ccb30b4 >2018-05-29 17:00:03,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:00:03,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:00:03,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:00:03,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4ccb30b4 >2018-05-29 17:00:03,475-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:00:03,488-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3915399e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:00:03,489-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3915399e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c910db0 >2018-05-29 17:00:03,491-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3915399e] FINISH, SetVdsStatusVDSCommand, log id: 6c910db0 >2018-05-29 17:00:03,497-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3915399e, Job ID: abc0a669-8a56-40f4-b275-fbce49505708, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:00:03,501-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 342355ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:00:03,503-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3915399e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 342355ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:00:03,515-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [6f64556] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:00:03,516-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [6f64556] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [55d06484] Autorecovering 2 hosts >2018-05-29 17:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [55d06484] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:05:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [49638486] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c8ab1c1 >2018-05-29 17:05:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [49638486] FINISH, SetVdsStatusVDSCommand, log id: 7c8ab1c1 >2018-05-29 17:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [49638486] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:05:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [49638486] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70659e6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ca5d426 >2018-05-29 17:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [70659e6a] FINISH, SetVdsStatusVDSCommand, log id: 2ca5d426 >2018-05-29 17:05:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [70659e6a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7e8ca2d4 >2018-05-29 17:05:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [70659e6a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7e8ca2d4 >2018-05-29 17:05:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:05:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [70659e6a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:05:01,170-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [495250cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1182974a >2018-05-29 17:05:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [495250cb] FINISH, GetHardwareInfoVDSCommand, log id: 1182974a >2018-05-29 17:05:01,186-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [34464424] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:05:01,195-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [306cfa28] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:05:01,201-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [51a7edbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:05:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [51a7edbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 406006e7 >2018-05-29 17:05:01,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:05:01,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [51a7edbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 406006e7 >2018-05-29 17:05:01,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:01,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:01,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:01,641-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:01,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:01,645-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [51a7edbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:01,649-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [51a7edbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:05:01,651-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [51a7edbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51a7edbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:05:01,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3a25327d >2018-05-29 17:05:01,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:05:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:05:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:05:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3a25327d >2018-05-29 17:05:01,757-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:05:01,770-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4a99bd47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:05:01,771-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4a99bd47] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43ac1d14 >2018-05-29 17:05:01,774-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4a99bd47] FINISH, SetVdsStatusVDSCommand, log id: 43ac1d14 >2018-05-29 17:05:01,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a99bd47, Job ID: 0b254bb3-e56c-4fed-b957-ed70c47a2597, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:05:01,810-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 306cfa28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:05:01,812-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4a99bd47] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 306cfa28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:05:01,823-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2d72159] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:05:01,824-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2d72159] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:05:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2f7b853b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 476873fb >2018-05-29 17:05:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2f7b853b] FINISH, GetHardwareInfoVDSCommand, log id: 476873fb >2018-05-29 17:05:03,182-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3c440e13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:05:03,187-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7059e04a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:05:03,192-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [37c9ccf9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:05:03,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [37c9ccf9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39e10eac >2018-05-29 17:05:03,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:05:03,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [37c9ccf9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39e10eac >2018-05-29 17:05:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:03,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:03,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:03,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:03,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:05:03,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [37c9ccf9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:05:03,609-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [37c9ccf9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:05:03,612-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [37c9ccf9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37c9ccf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:05:03,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3eaea35 >2018-05-29 17:05:03,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:05:03,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:05:03,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:05:03,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3eaea35 >2018-05-29 17:05:03,717-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:05:03,730-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [27475e5e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:05:03,731-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27475e5e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 425a98dc >2018-05-29 17:05:03,733-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27475e5e] FINISH, SetVdsStatusVDSCommand, log id: 425a98dc >2018-05-29 17:05:03,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27475e5e, Job ID: ada62e36-babc-461c-ab59-cc1a394d3e30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:05:03,771-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7059e04a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:05:03,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27475e5e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7059e04a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:05:03,784-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3b9dc3f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:05:03,785-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3b9dc3f1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3910ca68] Autorecovering 2 hosts >2018-05-29 17:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3910ca68] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:10:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [72700d72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48354030 >2018-05-29 17:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [72700d72] FINISH, SetVdsStatusVDSCommand, log id: 48354030 >2018-05-29 17:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [72700d72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:10:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [72700d72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:10:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6fcbe42 >2018-05-29 17:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] FINISH, SetVdsStatusVDSCommand, log id: 6fcbe42 >2018-05-29 17:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ff1eab9 >2018-05-29 17:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [13d8a87a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ff1eab9 >2018-05-29 17:10:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:10:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [13d8a87a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:10:01,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [632f1043] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f34f211 >2018-05-29 17:10:01,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [632f1043] FINISH, GetHardwareInfoVDSCommand, log id: 1f34f211 >2018-05-29 17:10:01,372-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6a959407] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:10:01,381-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3ce63245] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:10:01,386-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6852eed9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:10:01,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6852eed9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 488c1289 >2018-05-29 17:10:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:10:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6852eed9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 488c1289 >2018-05-29 17:10:01,777-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:01,778-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:01,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:01,782-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:01,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:01,787-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6852eed9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:01,790-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6852eed9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:10:01,793-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6852eed9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6852eed9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:10:01,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4c23b5c9 >2018-05-29 17:10:01,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:10:01,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:10:01,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:10:01,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4c23b5c9 >2018-05-29 17:10:01,893-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:10:01,907-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [168399fe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:10:01,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [168399fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 580b1f14 >2018-05-29 17:10:01,911-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [168399fe] FINISH, SetVdsStatusVDSCommand, log id: 580b1f14 >2018-05-29 17:10:01,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 168399fe, Job ID: 520ebbbb-4562-4172-bae7-1a8391c443e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:10:01,947-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3ce63245, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:10:01,949-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [168399fe] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3ce63245, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:10:01,960-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4cc43fe8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:10:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4cc43fe8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:10:02,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [343f98b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 54c1e6c4 >2018-05-29 17:10:02,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [343f98b] FINISH, GetHardwareInfoVDSCommand, log id: 54c1e6c4 >2018-05-29 17:10:02,698-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [70afd45f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:10:02,703-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2cfe1494] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:10:02,708-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4072a2e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:10:02,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4072a2e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69fc102f >2018-05-29 17:10:03,088-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:10:03,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [4072a2e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69fc102f >2018-05-29 17:10:03,092-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:03,093-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:03,097-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:03,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:10:03,102-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [4072a2e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:10:03,106-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [4072a2e2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:10:03,109-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4072a2e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4072a2e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:10:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 71e7282b >2018-05-29 17:10:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:10:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:10:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:10:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 71e7282b >2018-05-29 17:10:03,208-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:10:03,223-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [eef516e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:10:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [eef516e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f7091dd >2018-05-29 17:10:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [eef516e] FINISH, SetVdsStatusVDSCommand, log id: 1f7091dd >2018-05-29 17:10:03,232-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: eef516e, Job ID: b779860d-1cf8-47de-b77f-87f559776da7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:10:03,237-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2cfe1494, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:10:03,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [eef516e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2cfe1494, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:10:03,250-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4996ccc7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:10:03,251-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4996ccc7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [21d7e152] Autorecovering 2 hosts >2018-05-29 17:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [21d7e152] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:15:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [19cf5177] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3de7be04 >2018-05-29 17:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [19cf5177] FINISH, SetVdsStatusVDSCommand, log id: 3de7be04 >2018-05-29 17:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [19cf5177] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:15:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [19cf5177] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7029d6de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d76dff4 >2018-05-29 17:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7029d6de] FINISH, SetVdsStatusVDSCommand, log id: 2d76dff4 >2018-05-29 17:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7029d6de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 248d2415 >2018-05-29 17:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7029d6de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 248d2415 >2018-05-29 17:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7029d6de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:15:00,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38dc62de] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c152f47 >2018-05-29 17:15:00,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38dc62de] FINISH, GetHardwareInfoVDSCommand, log id: c152f47 >2018-05-29 17:15:00,958-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2997745b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:15:00,967-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2da2e598] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:15:00,973-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [627e752] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:15:00,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [627e752] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1c503ce9 >2018-05-29 17:15:01,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:15:01,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [627e752] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1c503ce9 >2018-05-29 17:15:01,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:01,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:01,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:01,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:01,368-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:01,369-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [627e752] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:01,372-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [627e752] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:15:01,374-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [627e752] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 627e752, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:15:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 632098d1 >2018-05-29 17:15:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:15:01,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:15:01,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:15:01,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 632098d1 >2018-05-29 17:15:01,480-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:15:01,494-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5ea139f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:15:01,495-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ea139f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43945540 >2018-05-29 17:15:01,498-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ea139f3] FINISH, SetVdsStatusVDSCommand, log id: 43945540 >2018-05-29 17:15:01,530-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ea139f3, Job ID: 1e5ddabb-5c7c-46d3-af44-11ad3f276464, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:15:01,535-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2da2e598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:15:01,537-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5ea139f3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2da2e598, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:15:01,548-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [11ce3fb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:15:01,549-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [11ce3fb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:15:02,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2cb0d106] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 25bc382 >2018-05-29 17:15:02,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2cb0d106] FINISH, GetHardwareInfoVDSCommand, log id: 25bc382 >2018-05-29 17:15:02,586-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4c4b9cd6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:15:02,590-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [74e99cf0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:15:02,596-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e0aa180] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:15:02,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e0aa180] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34c8ff17 >2018-05-29 17:15:02,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:15:02,978-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e0aa180] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34c8ff17 >2018-05-29 17:15:02,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:02,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:02,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:15:02,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e0aa180] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:15:02,995-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e0aa180] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:15:02,997-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e0aa180] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e0aa180, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:15:03,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5e34dc99 >2018-05-29 17:15:03,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:15:03,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:15:03,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:15:03,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5e34dc99 >2018-05-29 17:15:03,097-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:15:03,111-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c88f790] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:15:03,112-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c88f790] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43574dc8 >2018-05-29 17:15:03,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c88f790] FINISH, SetVdsStatusVDSCommand, log id: 43574dc8 >2018-05-29 17:15:03,120-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c88f790, Job ID: f38d0680-08bb-456b-a09f-7c59b31bcbd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:15:03,125-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74e99cf0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:15:03,127-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c88f790] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74e99cf0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:15:03,138-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [747279f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:15:03,139-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [747279f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51ffeec0] Autorecovering 2 hosts >2018-05-29 17:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51ffeec0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3e4dea9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c2ca86c >2018-05-29 17:20:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3e4dea9a] FINISH, SetVdsStatusVDSCommand, log id: 5c2ca86c >2018-05-29 17:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3e4dea9a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:20:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3e4dea9a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [118f1326] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4570c734 >2018-05-29 17:20:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [118f1326] FINISH, SetVdsStatusVDSCommand, log id: 4570c734 >2018-05-29 17:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [118f1326] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4af9f2ae >2018-05-29 17:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [118f1326] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4af9f2ae >2018-05-29 17:20:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [118f1326] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:20:00,751-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6263f6f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: f4a9cf2 >2018-05-29 17:20:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6263f6f4] FINISH, GetHardwareInfoVDSCommand, log id: f4a9cf2 >2018-05-29 17:20:00,765-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7b348628] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:20:00,773-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3f3da94e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:20:00,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [372ffecd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:20:00,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [372ffecd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 625201a >2018-05-29 17:20:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:20:01,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [372ffecd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 625201a >2018-05-29 17:20:01,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:01,176-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:01,180-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:01,180-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:01,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:01,185-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [372ffecd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:01,188-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [372ffecd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:20:01,191-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [372ffecd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 372ffecd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:20:01,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 27f5b8b0 >2018-05-29 17:20:01,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:20:01,292-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:20:01,292-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:20:01,292-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:20:01,293-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:20:01,293-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 27f5b8b0 >2018-05-29 17:20:01,293-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:20:01,306-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [75d3e20a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:20:01,307-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [75d3e20a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79e0826 >2018-05-29 17:20:01,310-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [75d3e20a] FINISH, SetVdsStatusVDSCommand, log id: 79e0826 >2018-05-29 17:20:01,341-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 75d3e20a, Job ID: cb408361-699f-461d-b984-319f5e0618b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:20:01,346-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3f3da94e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:20:01,348-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [75d3e20a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3f3da94e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:20:01,360-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2b13373b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:20:01,361-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2b13373b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:20:02,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [89c4559] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 644c721a >2018-05-29 17:20:02,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [89c4559] FINISH, GetHardwareInfoVDSCommand, log id: 644c721a >2018-05-29 17:20:02,286-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [77b0439] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:20:02,294-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [f85bef9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:20:02,300-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2c378018] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:20:02,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2c378018] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23df10a9 >2018-05-29 17:20:03,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:20:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2c378018] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23df10a9 >2018-05-29 17:20:03,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:03,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:03,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:03,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:03,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:20:03,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2c378018] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:20:03,697-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2c378018] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:20:03,700-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2c378018] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c378018, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:20:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 755bde48 >2018-05-29 17:20:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:20:03,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:20:03,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:20:03,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 755bde48 >2018-05-29 17:20:03,806-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:20:03,819-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1d21f687] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:20:03,821-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d21f687] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13aa60e7 >2018-05-29 17:20:03,823-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1d21f687] FINISH, SetVdsStatusVDSCommand, log id: 13aa60e7 >2018-05-29 17:20:03,854-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d21f687, Job ID: 1a07df34-a82e-4fc9-8cbc-85fb158a43fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:20:03,859-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f85bef9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:20:03,861-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1d21f687] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f85bef9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:20:03,872-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6f004968] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:20:03,873-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6f004968] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:25:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c21f8fe] Autorecovering 2 hosts >2018-05-29 17:25:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c21f8fe] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:25:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:25:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:25:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [79e1075d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ae92b1f >2018-05-29 17:25:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [79e1075d] FINISH, SetVdsStatusVDSCommand, log id: 5ae92b1f >2018-05-29 17:25:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [79e1075d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:25:00,052-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [79e1075d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:25:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [272c1f21] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54725e9f >2018-05-29 17:25:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [272c1f21] FINISH, SetVdsStatusVDSCommand, log id: 54725e9f >2018-05-29 17:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [272c1f21] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7cd9eafd >2018-05-29 17:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [272c1f21] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7cd9eafd >2018-05-29 17:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [272c1f21] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:25:01,555-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d19658c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d4d03ab >2018-05-29 17:25:01,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d19658c] FINISH, GetHardwareInfoVDSCommand, log id: d4d03ab >2018-05-29 17:25:01,570-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [36614150] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:25:01,574-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [27f1cfb6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:25:01,579-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6dae818f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:25:01,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6dae818f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 31ead308 >2018-05-29 17:25:01,960-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:25:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6dae818f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 31ead308 >2018-05-29 17:25:01,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [63cb55ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 53049b21 >2018-05-29 17:25:01,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [63cb55ea] FINISH, GetHardwareInfoVDSCommand, log id: 53049b21 >2018-05-29 17:25:01,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:01,965-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:01,970-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:01,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:01,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6dae818f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:01,976-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [903f75f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:25:01,979-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6dae818f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:25:01,980-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5dc66ca7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:25:01,982-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6dae818f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6dae818f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:25:01,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a8bd1d2 >2018-05-29 17:25:01,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:25:01,986-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [78cba89e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:25:01,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [78cba89e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5737ac7f >2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:25:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:25:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:25:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a8bd1d2 >2018-05-29 17:25:02,083-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:25:02,097-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [550aefcd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:25:02,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [550aefcd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54d2bd55 >2018-05-29 17:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [550aefcd] FINISH, SetVdsStatusVDSCommand, log id: 54d2bd55 >2018-05-29 17:25:02,132-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 550aefcd, Job ID: 91782706-c361-46e8-889d-dfc64038f7e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:25:02,136-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27f1cfb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:25:02,138-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [550aefcd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27f1cfb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:25:02,149-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2205ac88] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:25:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2205ac88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:25:02,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:25:02,369-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [78cba89e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5737ac7f >2018-05-29 17:25:02,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:02,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:02,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:02,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:25:02,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [78cba89e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:25:02,386-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [78cba89e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:25:02,389-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [78cba89e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 78cba89e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:25:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1fe4e5d >2018-05-29 17:25:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1fe4e5d >2018-05-29 17:25:02,490-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:25:02,503-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5b84f71f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:25:02,504-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5b84f71f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 719d905a >2018-05-29 17:25:02,506-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5b84f71f] FINISH, SetVdsStatusVDSCommand, log id: 719d905a >2018-05-29 17:25:02,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5b84f71f, Job ID: 9f9f56a8-0868-4864-af95-864e462ee682, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:25:02,516-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5dc66ca7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:25:02,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5b84f71f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5dc66ca7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:25:02,530-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [661c1fab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:25:02,531-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [661c1fab] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:30:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [532ef1f7] Autorecovering 2 hosts >2018-05-29 17:30:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [532ef1f7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:30:00,031-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:30:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:30:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7fafbd72] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5bf7337e >2018-05-29 17:30:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7fafbd72] FINISH, SetVdsStatusVDSCommand, log id: 5bf7337e >2018-05-29 17:30:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7fafbd72] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:30:00,045-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7fafbd72] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:30:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:30:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:30:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,058-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6a413878] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 749d6366 >2018-05-29 17:30:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6a413878] FINISH, SetVdsStatusVDSCommand, log id: 749d6366 >2018-05-29 17:30:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6a413878] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1082beb4 >2018-05-29 17:30:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6a413878] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1082beb4 >2018-05-29 17:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:30:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6a413878] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:30:02,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1b09a1a9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 240ddbd2 >2018-05-29 17:30:02,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [1b09a1a9] FINISH, GetHardwareInfoVDSCommand, log id: 240ddbd2 >2018-05-29 17:30:02,825-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [53df450a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:30:02,829-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7eaec964] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:30:02,835-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7ff0b433] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:30:02,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7ff0b433] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ab85a9c >2018-05-29 17:30:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1d0b17e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: fa6389a >2018-05-29 17:30:03,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1d0b17e7] FINISH, GetHardwareInfoVDSCommand, log id: fa6389a >2018-05-29 17:30:03,163-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [20425622] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:30:03,167-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2b17de51] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:30:03,173-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5e33e3cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:30:03,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5e33e3cc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 422930ef >2018-05-29 17:30:03,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:30:03,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5e33e3cc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 422930ef >2018-05-29 17:30:03,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:03,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:03,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:03,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:03,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:03,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5e33e3cc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:03,628-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5e33e3cc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:30:03,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5e33e3cc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e33e3cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:30:03,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fbc806b >2018-05-29 17:30:03,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:30:03,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:30:03,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:30:03,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fbc806b >2018-05-29 17:30:03,736-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:30:03,750-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [39c43a5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:30:03,751-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [39c43a5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: dd4bb1f >2018-05-29 17:30:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [39c43a5b] FINISH, SetVdsStatusVDSCommand, log id: dd4bb1f >2018-05-29 17:30:03,786-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39c43a5b, Job ID: 40ba19ca-1cce-48c3-9382-eeef71ff4d11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:30:03,791-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2b17de51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:30:03,793-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [39c43a5b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2b17de51, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:30:03,808-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48401a59] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:30:03,809-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48401a59] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:30:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:30:04,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [7ff0b433] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6ab85a9c >2018-05-29 17:30:04,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:04,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:04,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:04,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:04,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:30:04,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [7ff0b433] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:30:04,237-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [7ff0b433] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:30:04,240-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7ff0b433] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ff0b433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:30:04,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 824a683 >2018-05-29 17:30:04,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:30:04,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:30:04,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:30:04,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 824a683 >2018-05-29 17:30:04,340-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:30:04,353-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4f995eec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:30:04,354-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4f995eec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4560e92c >2018-05-29 17:30:04,356-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4f995eec] FINISH, SetVdsStatusVDSCommand, log id: 4560e92c >2018-05-29 17:30:04,362-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f995eec, Job ID: 5b1b2175-c620-4fde-bcb5-2773e47fbbab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:30:04,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7eaec964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:30:04,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4f995eec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7eaec964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:30:04,380-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3e0e2f15] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:30:04,381-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3e0e2f15] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c514688] Autorecovering 2 hosts >2018-05-29 17:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2c514688] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:35:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:35:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3ac65401] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7379ae95 >2018-05-29 17:35:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3ac65401] FINISH, SetVdsStatusVDSCommand, log id: 7379ae95 >2018-05-29 17:35:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3ac65401] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:35:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3ac65401] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:35:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:35:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:35:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7e61b152] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30d032c8 >2018-05-29 17:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7e61b152] FINISH, SetVdsStatusVDSCommand, log id: 30d032c8 >2018-05-29 17:35:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7e61b152] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7fb8e440 >2018-05-29 17:35:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7e61b152] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7fb8e440 >2018-05-29 17:35:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:35:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7e61b152] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:35:01,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [52c02ccf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 324556c8 >2018-05-29 17:35:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [52c02ccf] FINISH, GetHardwareInfoVDSCommand, log id: 324556c8 >2018-05-29 17:35:01,504-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [192078e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:35:01,512-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6b3ac934] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:35:01,517-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5e3a7f25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:35:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5e3a7f25] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5957bcfa >2018-05-29 17:35:01,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:35:01,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5e3a7f25] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5957bcfa >2018-05-29 17:35:01,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:01,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:01,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:01,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:01,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:01,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5e3a7f25] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:01,919-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5e3a7f25] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:35:01,922-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5e3a7f25] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e3a7f25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:35:01,928-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48074259 >2018-05-29 17:35:01,928-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:35:01,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2d33a2b7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 70dee6c0 >2018-05-29 17:35:01,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2d33a2b7] FINISH, GetHardwareInfoVDSCommand, log id: 70dee6c0 >2018-05-29 17:35:02,011-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [64af1bbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:35:02,016-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [63f6a33] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:35:02,021-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6d225339] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:35:02,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:35:02,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:35:02,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48074259 >2018-05-29 17:35:02,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:35:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6d225339] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41e27c62 >2018-05-29 17:35:02,041-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6bce060a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:35:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6bce060a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 301f275f >2018-05-29 17:35:02,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6bce060a] FINISH, SetVdsStatusVDSCommand, log id: 301f275f >2018-05-29 17:35:02,076-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bce060a, Job ID: 4c2655b9-6a94-4498-a5d3-f0117b6d4896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:35:02,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b3ac934, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:35:02,083-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6bce060a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b3ac934, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:35:02,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [110dde19] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:35:02,095-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [110dde19] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:35:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:35:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6d225339] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41e27c62 >2018-05-29 17:35:02,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:02,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:02,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:02,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:02,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:35:02,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6d225339] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:35:02,418-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6d225339] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:35:02,420-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6d225339] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d225339, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:35:02,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f6a6a6e >2018-05-29 17:35:02,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:35:02,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:35:02,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:35:02,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f6a6a6e >2018-05-29 17:35:02,523-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:35:02,536-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [757662af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:35:02,538-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [757662af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 370ca352 >2018-05-29 17:35:02,540-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [757662af] FINISH, SetVdsStatusVDSCommand, log id: 370ca352 >2018-05-29 17:35:02,545-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 757662af, Job ID: 496932b1-8ed7-4cbe-8cd1-0d06375a52e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:35:02,550-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63f6a33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:35:02,552-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [757662af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63f6a33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:35:02,563-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [17c90764] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:35:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [17c90764] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4758d012] Autorecovering 2 hosts >2018-05-29 17:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4758d012] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:40:00,036-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c845482] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 44fdcb31 >2018-05-29 17:40:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c845482] FINISH, SetVdsStatusVDSCommand, log id: 44fdcb31 >2018-05-29 17:40:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c845482] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:40:00,050-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6c845482] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [187b6309] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2df88070 >2018-05-29 17:40:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [187b6309] FINISH, SetVdsStatusVDSCommand, log id: 2df88070 >2018-05-29 17:40:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [187b6309] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51ab559a >2018-05-29 17:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [187b6309] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51ab559a >2018-05-29 17:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [187b6309] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:40:00,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [8fae41c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 71457182 >2018-05-29 17:40:00,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [8fae41c] FINISH, GetHardwareInfoVDSCommand, log id: 71457182 >2018-05-29 17:40:00,232-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [669b945] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:40:00,240-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [cfcf71e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:40:00,245-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [28e912cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:40:00,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [28e912cf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b3a6b3a >2018-05-29 17:40:00,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:40:00,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [28e912cf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b3a6b3a >2018-05-29 17:40:00,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:00,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:00,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:00,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:00,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:00,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [28e912cf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:00,646-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [28e912cf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:40:00,649-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [28e912cf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 28e912cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:40:00,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9d5f80 >2018-05-29 17:40:00,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:40:00,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:40:00,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:40:00,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9d5f80 >2018-05-29 17:40:00,752-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:40:00,767-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [256e432b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:40:00,768-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [256e432b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5968ed04 >2018-05-29 17:40:00,770-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [256e432b] FINISH, SetVdsStatusVDSCommand, log id: 5968ed04 >2018-05-29 17:40:00,802-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 256e432b, Job ID: c3c265ed-fdda-4cff-9146-b3a3218927f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:40:00,807-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: cfcf71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:40:00,809-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [256e432b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: cfcf71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:40:00,820-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [281dfe9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:40:00,821-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [281dfe9d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:40:02,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [202766c4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c81ac6b >2018-05-29 17:40:02,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [202766c4] FINISH, GetHardwareInfoVDSCommand, log id: 5c81ac6b >2018-05-29 17:40:02,850-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [52f162f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:40:02,855-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [249e7524] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:40:02,860-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [29384491] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:40:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [29384491] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69908fa6 >2018-05-29 17:40:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:40:03,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [29384491] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69908fa6 >2018-05-29 17:40:03,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:03,248-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:03,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:03,253-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:03,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:40:03,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [29384491] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:40:03,260-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [29384491] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:40:03,263-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [29384491] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29384491, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:40:03,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b030275 >2018-05-29 17:40:03,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:40:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:40:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:40:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b030275 >2018-05-29 17:40:03,363-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:40:03,376-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2df7baee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:40:03,377-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7baee] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f4ee420 >2018-05-29 17:40:03,379-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7baee] FINISH, SetVdsStatusVDSCommand, log id: 6f4ee420 >2018-05-29 17:40:03,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2df7baee, Job ID: b785c13d-5352-4084-8eaf-61a49bb92d28, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:40:03,390-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 249e7524, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:40:03,392-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2df7baee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 249e7524, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:40:03,402-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3a1f5a0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:40:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3a1f5a0e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [65bcc42] Autorecovering 2 hosts >2018-05-29 17:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [65bcc42] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:45:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:45:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:45:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2e33be16] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18fde54d >2018-05-29 17:45:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2e33be16] FINISH, SetVdsStatusVDSCommand, log id: 18fde54d >2018-05-29 17:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2e33be16] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:45:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [2e33be16] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:45:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47002d6a >2018-05-29 17:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] FINISH, SetVdsStatusVDSCommand, log id: 47002d6a >2018-05-29 17:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4e566fe1 >2018-05-29 17:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [e82e4c6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4e566fe1 >2018-05-29 17:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:45:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [e82e4c6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:45:01,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [57deba71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50610bf4 >2018-05-29 17:45:01,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [57deba71] FINISH, GetHardwareInfoVDSCommand, log id: 50610bf4 >2018-05-29 17:45:01,091-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [440ec4b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:45:01,100-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [f6c57a9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:45:01,105-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [718fb07f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:45:01,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [718fb07f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 54457eed >2018-05-29 17:45:01,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [706f5176] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 581e045a >2018-05-29 17:45:01,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [706f5176] FINISH, GetHardwareInfoVDSCommand, log id: 581e045a >2018-05-29 17:45:01,441-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [30b203aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:45:01,446-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1100b670] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:45:01,451-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4b3a84eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:45:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4b3a84eb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b99d405 >2018-05-29 17:45:01,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:45:01,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [718fb07f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 54457eed >2018-05-29 17:45:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,513-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [718fb07f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,516-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [718fb07f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:45:01,518-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [718fb07f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 718fb07f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:45:01,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7574f3b9 >2018-05-29 17:45:01,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:45:01,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:45:01,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:45:01,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7574f3b9 >2018-05-29 17:45:01,624-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:45:01,637-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [32d4ff2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:45:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32d4ff2c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11b2e722 >2018-05-29 17:45:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [32d4ff2c] FINISH, SetVdsStatusVDSCommand, log id: 11b2e722 >2018-05-29 17:45:01,674-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32d4ff2c, Job ID: 5b805d7a-7dca-42a6-9074-d1b6b932cb5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:45:01,678-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f6c57a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:45:01,681-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32d4ff2c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f6c57a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:45:01,692-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5b637cc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:45:01,693-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5b637cc9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:45:01,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:45:01,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4b3a84eb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b99d405 >2018-05-29 17:45:01,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:45:01,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4b3a84eb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:45:01,840-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4b3a84eb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:45:01,843-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b3a84eb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4b3a84eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:45:01,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2a718dee >2018-05-29 17:45:01,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:45:01,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:45:01,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:45:01,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a718dee >2018-05-29 17:45:01,943-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:45:01,957-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1e71229f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:45:01,958-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1e71229f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ec10cb6 >2018-05-29 17:45:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1e71229f] FINISH, SetVdsStatusVDSCommand, log id: ec10cb6 >2018-05-29 17:45:01,995-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e71229f, Job ID: c12f5ce0-9d48-4aa4-9606-32b7e96eafce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:45:01,999-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1100b670, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:45:02,001-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1e71229f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1100b670, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:45:02,013-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [250aa785] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:45:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [250aa785] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2c8de916] Autorecovering 2 hosts >2018-05-29 17:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2c8de916] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [bc636f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d027e39 >2018-05-29 17:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [bc636f1] FINISH, SetVdsStatusVDSCommand, log id: d027e39 >2018-05-29 17:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [bc636f1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:50:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [bc636f1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ae9b7b5 >2018-05-29 17:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] FINISH, SetVdsStatusVDSCommand, log id: 1ae9b7b5 >2018-05-29 17:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2c38c075 >2018-05-29 17:50:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7cdaac7c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2c38c075 >2018-05-29 17:50:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:50:00,100-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7cdaac7c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 7cdaac7c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 17:50:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7cdaac7c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:50:02,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [33d96b92] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23540cdb >2018-05-29 17:50:02,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [33d96b92] FINISH, GetHardwareInfoVDSCommand, log id: 23540cdb >2018-05-29 17:50:02,331-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3d1511fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:50:02,339-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [73b2ed16] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:50:02,343-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ff437d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:50:02,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ff437d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a73f122 >2018-05-29 17:50:02,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [29fe6b34] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ef344a2 >2018-05-29 17:50:02,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [29fe6b34] FINISH, GetHardwareInfoVDSCommand, log id: 1ef344a2 >2018-05-29 17:50:02,676-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [608bb065] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:50:02,681-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [43f11178] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:50:02,686-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e28b368] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:50:02,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e28b368] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c1c2a6c >2018-05-29 17:50:02,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:50:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5ff437d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a73f122 >2018-05-29 17:50:02,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:02,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:02,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:02,742-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:02,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:02,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5ff437d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:02,752-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5ff437d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:50:02,756-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5ff437d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ff437d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:50:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: dd6db2a >2018-05-29 17:50:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:50:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:50:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:50:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: dd6db2a >2018-05-29 17:50:02,858-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:50:02,872-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [31c278ad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:50:02,873-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [31c278ad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6472732e >2018-05-29 17:50:02,875-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [31c278ad] FINISH, SetVdsStatusVDSCommand, log id: 6472732e >2018-05-29 17:50:02,905-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31c278ad, Job ID: 549fc686-ea46-4573-ad9e-67114a1d353d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:50:02,909-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73b2ed16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:50:02,912-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [31c278ad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73b2ed16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:50:02,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [693520a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:50:02,924-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [693520a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:50:03,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:50:03,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [e28b368] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c1c2a6c >2018-05-29 17:50:03,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:03,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:03,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:03,073-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:03,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:50:03,077-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [e28b368] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:50:03,080-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [e28b368] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:50:03,082-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [e28b368] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: e28b368, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:50:03,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fa9978a >2018-05-29 17:50:03,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:50:03,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:50:03,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:50:03,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fa9978a >2018-05-29 17:50:03,186-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:50:03,200-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6feb6481] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:50:03,201-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6feb6481] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a7ec597 >2018-05-29 17:50:03,203-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6feb6481] FINISH, SetVdsStatusVDSCommand, log id: 6a7ec597 >2018-05-29 17:50:03,209-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6feb6481, Job ID: 41b9ccc6-cd1b-4195-b1a3-4050de7d8d4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:50:03,214-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 43f11178, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:50:03,217-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6feb6481] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 43f11178, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:50:03,227-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [fe64888] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:50:03,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [fe64888] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:54:43,136-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [2d9ed0ad] Attempting to update VMs/Templates Ovf. >2018-05-29 17:54:43,139-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 17:54:43,139-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 17:54:43,140-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:54:43,194-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 17:54:43,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 17:54:43,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 17:54:43,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 17:54:43,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 17:54:43,198-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 17:54:43,199-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [6b927e9b] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 17:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60f77b93] Autorecovering 2 hosts >2018-05-29 17:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [60f77b93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:55:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [77b99aaf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26057406 >2018-05-29 17:55:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [77b99aaf] FINISH, SetVdsStatusVDSCommand, log id: 26057406 >2018-05-29 17:55:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [77b99aaf] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 17:55:00,104-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [77b99aaf] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:55:00,110-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:55:00,116-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 17:55:00,116-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,116-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,118-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9c999a5 >2018-05-29 17:55:00,120-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] FINISH, SetVdsStatusVDSCommand, log id: 9c999a5 >2018-05-29 17:55:00,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79d0f0b4 >2018-05-29 17:55:00,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5e343b9b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79d0f0b4 >2018-05-29 17:55:00,125-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 17:55:00,126-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5e343b9b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 17:55:00,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [74b786b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23235ffd >2018-05-29 17:55:00,578-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [74b786b8] FINISH, GetHardwareInfoVDSCommand, log id: 23235ffd >2018-05-29 17:55:00,591-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7488fe8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:55:00,599-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [514e75de] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:55:00,606-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2792c42b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:55:00,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2792c42b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79d51b61 >2018-05-29 17:55:00,871-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3d3a5d5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 47ea087d >2018-05-29 17:55:00,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3d3a5d5c] FINISH, GetHardwareInfoVDSCommand, log id: 47ea087d >2018-05-29 17:55:00,886-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5bbfc60e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:55:00,890-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [319418ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:55:00,895-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7061978e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 17:55:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7061978e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a3154ec >2018-05-29 17:55:01,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:55:01,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2792c42b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79d51b61 >2018-05-29 17:55:01,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:01,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:01,053-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:01,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:01,058-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2792c42b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:01,060-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2792c42b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:55:01,063-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2792c42b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2792c42b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:55:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26669b27 >2018-05-29 17:55:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:55:01,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:55:01,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:55:01,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26669b27 >2018-05-29 17:55:01,163-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:55:01,177-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [44384b1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:55:01,178-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [44384b1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ecaa474 >2018-05-29 17:55:01,180-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [44384b1b] FINISH, SetVdsStatusVDSCommand, log id: 1ecaa474 >2018-05-29 17:55:01,188-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 44384b1b, Job ID: ac591d78-3629-4884-831e-4b944e4ff9cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:55:01,193-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 514e75de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:55:01,195-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44384b1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 514e75de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:55:01,207-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5d399415] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 17:55:01,208-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5d399415] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 17:55:02,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 17:55:02,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7061978e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a3154ec >2018-05-29 17:55:02,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:02,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:02,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:02,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:02,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 17:55:02,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7061978e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 17:55:02,290-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7061978e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 17:55:02,294-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7061978e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7061978e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 17:55:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c3b58b >2018-05-29 17:55:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:55:02,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 17:55:02,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 17:55:02,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: c3b58b >2018-05-29 17:55:02,394-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 17:55:02,409-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3ccb237d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:55:02,410-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ccb237d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 624f4b8 >2018-05-29 17:55:02,412-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3ccb237d] FINISH, SetVdsStatusVDSCommand, log id: 624f4b8 >2018-05-29 17:55:02,444-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3ccb237d, Job ID: a1411b9b-a881-4e0d-9273-28dff826d3b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 17:55:02,449-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 319418ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 17:55:02,451-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ccb237d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 319418ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 17:55:02,462-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [354728f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 17:55:02,463-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [354728f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [79e0ef4a] Autorecovering 2 hosts >2018-05-29 18:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [79e0ef4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:00:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [69815e56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32def646 >2018-05-29 18:00:00,083-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [69815e56] FINISH, SetVdsStatusVDSCommand, log id: 32def646 >2018-05-29 18:00:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,089-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69815e56] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 69815e56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 18:00:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [69815e56] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:00:00,090-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [69815e56] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:00:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77433c20 >2018-05-29 18:00:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] FINISH, SetVdsStatusVDSCommand, log id: 77433c20 >2018-05-29 18:00:00,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23a6fbcf >2018-05-29 18:00:00,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4fac15d7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23a6fbcf >2018-05-29 18:00:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:00:00,112-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4fac15d7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:00:01,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20593be9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1820f201 >2018-05-29 18:00:01,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [20593be9] FINISH, GetHardwareInfoVDSCommand, log id: 1820f201 >2018-05-29 18:00:01,978-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [ba9d38c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:00:01,987-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [319a5c07] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:00:01,992-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [138afedc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:00:02,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [138afedc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 763ab777 >2018-05-29 18:00:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:00:02,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [138afedc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 763ab777 >2018-05-29 18:00:02,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:02,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:02,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:02,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:02,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:02,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [138afedc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:02,401-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [138afedc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:00:02,404-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [138afedc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 138afedc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:00:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c46517e >2018-05-29 18:00:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:00:02,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:00:02,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:00:02,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c46517e >2018-05-29 18:00:02,510-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:00:02,525-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [1f48ba20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:00:02,526-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f48ba20] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 105772d8 >2018-05-29 18:00:02,528-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f48ba20] FINISH, SetVdsStatusVDSCommand, log id: 105772d8 >2018-05-29 18:00:02,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f48ba20, Job ID: 031eb7df-283f-40a0-8cef-5ce4e5e45767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:00:02,563-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 319a5c07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:00:02,566-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f48ba20] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 319a5c07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:00:02,577-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4398f53f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:00:02,579-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4398f53f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:00:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [718966cc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 197790b4 >2018-05-29 18:00:03,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [718966cc] FINISH, GetHardwareInfoVDSCommand, log id: 197790b4 >2018-05-29 18:00:03,230-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [186cb42b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:00:03,238-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1e03a96b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:00:03,244-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7ccc71d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:00:03,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7ccc71d1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 56eb91f9 >2018-05-29 18:00:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:00:03,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7ccc71d1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 56eb91f9 >2018-05-29 18:00:03,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:03,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:03,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:03,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:03,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:00:03,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7ccc71d1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:00:03,647-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7ccc71d1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:00:03,650-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7ccc71d1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ccc71d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:00:03,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5eb61d93 >2018-05-29 18:00:03,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:00:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:00:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:00:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5eb61d93 >2018-05-29 18:00:03,753-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:00:03,769-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5d7b44a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:00:03,770-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5d7b44a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c78eaa8 >2018-05-29 18:00:03,773-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5d7b44a5] FINISH, SetVdsStatusVDSCommand, log id: c78eaa8 >2018-05-29 18:00:03,808-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5d7b44a5, Job ID: 9096be2d-e080-40e8-90c5-397c52773c2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:00:03,813-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1e03a96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:00:03,815-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5d7b44a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1e03a96b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:00:03,828-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6a227fe2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:00:03,829-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6a227fe2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3b9555b7] Autorecovering 2 hosts >2018-05-29 18:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3b9555b7] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:05:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:05:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5f9d1284] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5204f7a9 >2018-05-29 18:05:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5f9d1284] FINISH, SetVdsStatusVDSCommand, log id: 5204f7a9 >2018-05-29 18:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [5f9d1284] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:05:00,072-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5f9d1284] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:05:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:05:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:05:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d5ed2d4 >2018-05-29 18:05:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] FINISH, SetVdsStatusVDSCommand, log id: d5ed2d4 >2018-05-29 18:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 647a5e8f >2018-05-29 18:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [7424ce9a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 647a5e8f >2018-05-29 18:05:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:05:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7424ce9a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:05:01,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [502e1f91] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2f82ecba >2018-05-29 18:05:01,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [502e1f91] FINISH, GetHardwareInfoVDSCommand, log id: 2f82ecba >2018-05-29 18:05:01,523-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [328ad08e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:05:01,528-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5083a6ed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:05:01,533-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [287eae44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:05:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [287eae44] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7861072d >2018-05-29 18:05:01,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:05:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [287eae44] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7861072d >2018-05-29 18:05:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:01,906-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:01,910-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:01,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:01,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:01,915-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [287eae44] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:01,918-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [287eae44] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:05:01,921-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [287eae44] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 287eae44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:05:01,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fcea265 >2018-05-29 18:05:01,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:05:02,020-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:05:02,020-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:05:02,020-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fcea265 >2018-05-29 18:05:02,020-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:05:02,034-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1b315668] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:05:02,035-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b315668] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b625226 >2018-05-29 18:05:02,037-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1b315668] FINISH, SetVdsStatusVDSCommand, log id: 1b625226 >2018-05-29 18:05:02,043-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b315668, Job ID: b79e59d2-4585-4e40-aced-df9f141fbbe7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:05:02,048-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5083a6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:05:02,050-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1b315668] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5083a6ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:05:02,061-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4f86a98] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:05:02,062-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4f86a98] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:05:03,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10866b04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1eb1c87b >2018-05-29 18:05:03,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10866b04] FINISH, GetHardwareInfoVDSCommand, log id: 1eb1c87b >2018-05-29 18:05:03,215-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7f403594] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:05:03,220-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7b010592] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:05:03,225-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [76b9046b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:05:03,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [76b9046b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 739849a1 >2018-05-29 18:05:03,615-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:05:03,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [76b9046b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 739849a1 >2018-05-29 18:05:03,619-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:03,620-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:03,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:03,625-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:05:03,630-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [76b9046b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:05:03,633-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [76b9046b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:05:03,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [76b9046b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76b9046b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:05:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cf78534 >2018-05-29 18:05:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:05:03,738-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:05:03,738-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:05:03,738-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cf78534 >2018-05-29 18:05:03,738-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:05:03,751-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [b98bfb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:05:03,753-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [b98bfb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18afd2b5 >2018-05-29 18:05:03,755-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [b98bfb1] FINISH, SetVdsStatusVDSCommand, log id: 18afd2b5 >2018-05-29 18:05:03,761-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b98bfb1, Job ID: a357ffcc-04bc-400f-b85b-76a4535d9ec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:05:03,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b010592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:05:03,768-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [b98bfb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b010592, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:05:03,780-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7706ca68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:05:03,781-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7706ca68] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3d0b0a10] Autorecovering 2 hosts >2018-05-29 18:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3d0b0a10] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7cf654f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6981f238 >2018-05-29 18:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7cf654f7] FINISH, SetVdsStatusVDSCommand, log id: 6981f238 >2018-05-29 18:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7cf654f7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:10:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7cf654f7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:10:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [528f3189] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60143544 >2018-05-29 18:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [528f3189] FINISH, SetVdsStatusVDSCommand, log id: 60143544 >2018-05-29 18:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [528f3189] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: b4f5b82 >2018-05-29 18:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [528f3189] FINISH, SetHaMaintenanceModeVDSCommand, log id: b4f5b82 >2018-05-29 18:10:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [528f3189] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:10:01,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [30583684] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 632bce3a >2018-05-29 18:10:01,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [30583684] FINISH, GetHardwareInfoVDSCommand, log id: 632bce3a >2018-05-29 18:10:01,552-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [258ec9c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:10:01,560-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4946a533] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:10:01,566-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [52ba9ad8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:10:01,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [52ba9ad8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4c632207 >2018-05-29 18:10:01,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:10:01,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [52ba9ad8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4c632207 >2018-05-29 18:10:01,960-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:01,961-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:01,965-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:01,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:01,970-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [52ba9ad8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:01,973-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [52ba9ad8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:10:01,975-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [52ba9ad8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52ba9ad8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:10:01,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75e40713 >2018-05-29 18:10:01,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:10:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:10:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:10:02,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75e40713 >2018-05-29 18:10:02,079-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:10:02,092-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [20de6c01] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:10:02,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [20de6c01] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19029b5a >2018-05-29 18:10:02,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [20de6c01] FINISH, SetVdsStatusVDSCommand, log id: 19029b5a >2018-05-29 18:10:02,127-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20de6c01, Job ID: fc458559-f2ab-4b28-a865-3a19c63c433b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:10:02,132-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4946a533, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:10:02,134-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20de6c01] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4946a533, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:10:02,145-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [40fdaaaf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:10:02,146-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [40fdaaaf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:10:02,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [454ae2b8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7186aa0d >2018-05-29 18:10:02,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [454ae2b8] FINISH, GetHardwareInfoVDSCommand, log id: 7186aa0d >2018-05-29 18:10:02,775-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [307ea221] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:10:02,779-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [39317f9b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:10:02,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5cf76ec0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:10:02,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5cf76ec0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 473d80e7 >2018-05-29 18:10:03,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:10:03,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5cf76ec0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 473d80e7 >2018-05-29 18:10:03,164-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:03,165-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:03,169-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:03,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:10:03,174-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5cf76ec0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:10:03,177-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5cf76ec0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:10:03,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5cf76ec0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cf76ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:10:03,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3857c5d3 >2018-05-29 18:10:03,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:10:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:10:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:10:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3857c5d3 >2018-05-29 18:10:03,281-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:10:03,294-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [72114d4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:10:03,295-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [72114d4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3275f4de >2018-05-29 18:10:03,297-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [72114d4f] FINISH, SetVdsStatusVDSCommand, log id: 3275f4de >2018-05-29 18:10:03,303-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72114d4f, Job ID: ca215b4b-b84c-44c5-9ef5-5d49fc885971, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:10:03,308-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 39317f9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:10:03,310-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [72114d4f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 39317f9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:10:03,320-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [673958c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:10:03,321-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [673958c2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1023ff79] Autorecovering 2 hosts >2018-05-29 18:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1023ff79] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:15:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ad83f66] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55da19a9 >2018-05-29 18:15:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3ad83f66] FINISH, SetVdsStatusVDSCommand, log id: 55da19a9 >2018-05-29 18:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3ad83f66] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:15:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3ad83f66] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:15:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74b035e1 >2018-05-29 18:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] FINISH, SetVdsStatusVDSCommand, log id: 74b035e1 >2018-05-29 18:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 63754fad >2018-05-29 18:15:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2cb13f87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 63754fad >2018-05-29 18:15:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:15:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2cb13f87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:15:00,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [39e18ff8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5b89b23c >2018-05-29 18:15:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [39e18ff8] FINISH, GetHardwareInfoVDSCommand, log id: 5b89b23c >2018-05-29 18:15:01,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4b4cb38c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:15:01,009-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [616800e8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:15:01,014-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [536ee703] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:15:01,021-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [536ee703] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23081148 >2018-05-29 18:15:02,387-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:15:02,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [536ee703] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23081148 >2018-05-29 18:15:02,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:02,392-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:02,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:02,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:02,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [536ee703] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:02,405-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [536ee703] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:15:02,408-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [536ee703] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 536ee703, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:15:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 253eb1d2 >2018-05-29 18:15:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:15:02,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:15:02,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:15:02,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 253eb1d2 >2018-05-29 18:15:02,508-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:15:02,522-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4119140c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:15:02,523-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4119140c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10e600b3 >2018-05-29 18:15:02,526-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4119140c] FINISH, SetVdsStatusVDSCommand, log id: 10e600b3 >2018-05-29 18:15:02,556-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4119140c, Job ID: a684a8b5-b6e8-4568-b337-e0d5c6d1d8fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:15:02,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 616800e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:15:02,563-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4119140c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 616800e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:15:02,573-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [35f216a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:15:02,574-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [35f216a2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:15:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3139ab6c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 38a94cc2 >2018-05-29 18:15:03,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3139ab6c] FINISH, GetHardwareInfoVDSCommand, log id: 38a94cc2 >2018-05-29 18:15:03,121-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [71ece68] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:15:03,125-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [41964c19] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:15:03,130-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [397c1340] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:15:03,136-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [397c1340] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4629bb8b >2018-05-29 18:15:03,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:15:03,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [397c1340] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4629bb8b >2018-05-29 18:15:03,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:03,522-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:03,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:03,526-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:03,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:15:03,531-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [397c1340] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:15:03,533-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [397c1340] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:15:03,536-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [397c1340] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 397c1340, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:15:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53848e24 >2018-05-29 18:15:03,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:15:03,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:15:03,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:15:03,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53848e24 >2018-05-29 18:15:03,637-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:15:03,650-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2d51b87d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:15:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d51b87d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b28d56 >2018-05-29 18:15:03,653-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d51b87d] FINISH, SetVdsStatusVDSCommand, log id: 43b28d56 >2018-05-29 18:15:03,659-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2d51b87d, Job ID: d75c5058-cd28-41d7-8d5d-2af2915ab3b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:15:03,664-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41964c19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:15:03,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2d51b87d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41964c19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:15:03,677-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [c0cc647] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:15:03,678-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [c0cc647] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:20:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5074d680] Autorecovering 2 hosts >2018-05-29 18:20:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5074d680] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ca2b874] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60ac05bc >2018-05-29 18:20:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5ca2b874] FINISH, SetVdsStatusVDSCommand, log id: 60ac05bc >2018-05-29 18:20:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5ca2b874] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:20:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5ca2b874] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:20:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:20:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 152bb54c >2018-05-29 18:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] FINISH, SetVdsStatusVDSCommand, log id: 152bb54c >2018-05-29 18:20:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 391791d9 >2018-05-29 18:20:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [713ae1ac] FINISH, SetHaMaintenanceModeVDSCommand, log id: 391791d9 >2018-05-29 18:20:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:20:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [713ae1ac] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:20:00,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [27c4da09] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7ae44862 >2018-05-29 18:20:00,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [27c4da09] FINISH, GetHardwareInfoVDSCommand, log id: 7ae44862 >2018-05-29 18:20:00,358-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [64772df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:20:00,363-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [60544686] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:20:00,369-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [344ce3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:20:00,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [344ce3b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c1d136f >2018-05-29 18:20:00,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:20:00,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [344ce3b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c1d136f >2018-05-29 18:20:00,758-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:00,759-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:00,762-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:00,763-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [344ce3b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:00,770-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [344ce3b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:20:00,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [344ce3b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 344ce3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:20:00,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36911c36 >2018-05-29 18:20:00,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:20:00,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:20:00,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:20:00,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36911c36 >2018-05-29 18:20:00,877-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:20:00,890-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [16aa4af3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:20:00,891-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [16aa4af3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cc4953b >2018-05-29 18:20:00,894-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [16aa4af3] FINISH, SetVdsStatusVDSCommand, log id: 3cc4953b >2018-05-29 18:20:00,902-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16aa4af3, Job ID: 6238b244-6aec-4cac-8f34-085bb1e5e386, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:20:00,907-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60544686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:20:00,909-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [16aa4af3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60544686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:20:00,920-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [364bae94] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:20:00,921-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [364bae94] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:20:01,283-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [37539e8a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 323ba2f8 >2018-05-29 18:20:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [37539e8a] FINISH, GetHardwareInfoVDSCommand, log id: 323ba2f8 >2018-05-29 18:20:01,297-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [587cfc0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:20:01,302-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [52b89977] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:20:01,307-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4f14598d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:20:01,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4f14598d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64d64a9b >2018-05-29 18:20:01,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:20:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4f14598d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64d64a9b >2018-05-29 18:20:01,694-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:01,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:01,698-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:01,699-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:20:01,704-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4f14598d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:20:01,706-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4f14598d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:20:01,709-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f14598d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f14598d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:20:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b237949 >2018-05-29 18:20:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:20:01,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:20:01,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:20:01,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b237949 >2018-05-29 18:20:01,810-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:20:01,823-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [56a30e48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:20:01,824-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [56a30e48] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27237d2b >2018-05-29 18:20:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [56a30e48] FINISH, SetVdsStatusVDSCommand, log id: 27237d2b >2018-05-29 18:20:01,831-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56a30e48, Job ID: c601316f-6872-4ffd-a5a1-e095e4f40a99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:20:01,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52b89977, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:20:01,837-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [56a30e48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52b89977, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:20:01,848-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6c21268e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:20:01,849-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6c21268e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79a15e26] Autorecovering 2 hosts >2018-05-29 18:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79a15e26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:25:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [125e5cac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 82190f7 >2018-05-29 18:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [125e5cac] FINISH, SetVdsStatusVDSCommand, log id: 82190f7 >2018-05-29 18:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [125e5cac] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:25:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [125e5cac] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1804dcab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28b616d3 >2018-05-29 18:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1804dcab] FINISH, SetVdsStatusVDSCommand, log id: 28b616d3 >2018-05-29 18:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1804dcab] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 293f436e >2018-05-29 18:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [1804dcab] FINISH, SetHaMaintenanceModeVDSCommand, log id: 293f436e >2018-05-29 18:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1804dcab] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:25:01,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2366417a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 130b52c3 >2018-05-29 18:25:01,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2366417a] FINISH, GetHardwareInfoVDSCommand, log id: 130b52c3 >2018-05-29 18:25:01,666-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6fb87457] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:25:01,675-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [17dcf698] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:25:01,680-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [369f0dbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:25:01,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [369f0dbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f73013b >2018-05-29 18:25:02,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:25:02,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [369f0dbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f73013b >2018-05-29 18:25:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,071-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [369f0dbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,083-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [369f0dbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:25:02,086-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [369f0dbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 369f0dbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:25:02,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2074a25a >2018-05-29 18:25:02,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:25:02,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:25:02,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:25:02,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2074a25a >2018-05-29 18:25:02,195-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:25:02,208-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [8522658] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:25:02,209-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [8522658] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38111abe >2018-05-29 18:25:02,211-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [8522658] FINISH, SetVdsStatusVDSCommand, log id: 38111abe >2018-05-29 18:25:02,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8522658, Job ID: 1372909b-f997-4282-85ce-8ce1ebd61a0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:25:02,248-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17dcf698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:25:02,251-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8522658] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17dcf698, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:25:02,261-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [59e52237] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:25:02,262-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [59e52237] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:25:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7eb5975f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 389f16fe >2018-05-29 18:25:02,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7eb5975f] FINISH, GetHardwareInfoVDSCommand, log id: 389f16fe >2018-05-29 18:25:02,519-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6e03177c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:25:02,552-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7f72dfd3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:25:02,557-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [593dee26] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:25:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [593dee26] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2f4f9a08 >2018-05-29 18:25:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:25:02,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [593dee26] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2f4f9a08 >2018-05-29 18:25:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:25:02,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [593dee26] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:25:02,957-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [593dee26] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:25:02,959-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [593dee26] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 593dee26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:25:02,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ef8c042 >2018-05-29 18:25:02,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:25:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:25:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:25:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: ef8c042 >2018-05-29 18:25:03,056-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:25:03,069-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [34775689] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:25:03,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [34775689] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20d3988f >2018-05-29 18:25:03,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [34775689] FINISH, SetVdsStatusVDSCommand, log id: 20d3988f >2018-05-29 18:25:03,078-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34775689, Job ID: cf6b40da-cd48-46c7-b012-43f5f56582c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:25:03,082-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f72dfd3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:25:03,084-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [34775689] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f72dfd3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:25:03,095-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6850579b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:25:03,096-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6850579b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d93462] Autorecovering 2 hosts >2018-05-29 18:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d93462] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7f529150] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 749ef548 >2018-05-29 18:30:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7f529150] FINISH, SetVdsStatusVDSCommand, log id: 749ef548 >2018-05-29 18:30:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7f529150] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:30:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7f529150] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:30:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:30:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:30:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [589e2afc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3892098c >2018-05-29 18:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [589e2afc] FINISH, SetVdsStatusVDSCommand, log id: 3892098c >2018-05-29 18:30:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [589e2afc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d7d20ed >2018-05-29 18:30:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [589e2afc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d7d20ed >2018-05-29 18:30:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:30:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [589e2afc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:30:00,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [b1bc22e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33a61ac9 >2018-05-29 18:30:00,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [b1bc22e] FINISH, GetHardwareInfoVDSCommand, log id: 33a61ac9 >2018-05-29 18:30:00,712-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [20f766f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:30:00,720-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [315f33d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:30:00,726-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [663e8ea9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:30:00,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [663e8ea9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65e3ecc >2018-05-29 18:30:01,117-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:30:01,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [663e8ea9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65e3ecc >2018-05-29 18:30:01,121-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:01,122-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:01,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:01,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:01,130-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [663e8ea9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:01,133-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [663e8ea9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:30:01,136-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [663e8ea9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 663e8ea9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:30:01,142-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 44e5f48a >2018-05-29 18:30:01,142-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:30:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:30:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:30:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 44e5f48a >2018-05-29 18:30:01,240-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:30:01,253-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1d6f39db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:30:01,254-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1d6f39db] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6051c7fc >2018-05-29 18:30:01,256-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1d6f39db] FINISH, SetVdsStatusVDSCommand, log id: 6051c7fc >2018-05-29 18:30:01,287-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d6f39db, Job ID: 97024c13-3cf1-44d9-a11e-d42d80d68577, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:30:01,291-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 315f33d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:30:01,293-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1d6f39db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 315f33d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:30:01,303-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [28301316] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:30:01,304-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [28301316] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:30:02,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1c100426] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 559bc929 >2018-05-29 18:30:02,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1c100426] FINISH, GetHardwareInfoVDSCommand, log id: 559bc929 >2018-05-29 18:30:02,982-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [37ef17f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:30:02,987-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [36c8bca0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:30:02,992-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [12a5c75a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:30:02,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [12a5c75a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 216c1ae7 >2018-05-29 18:30:03,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:30:03,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [12a5c75a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 216c1ae7 >2018-05-29 18:30:03,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:03,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:03,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:03,374-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:03,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:30:03,379-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [12a5c75a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:30:03,381-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [12a5c75a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:30:03,384-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [12a5c75a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12a5c75a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:30:03,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fb99575 >2018-05-29 18:30:03,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:30:03,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:30:03,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:30:03,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fb99575 >2018-05-29 18:30:03,484-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:30:03,497-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6637cf84] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:30:03,498-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6637cf84] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1319d341 >2018-05-29 18:30:03,500-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6637cf84] FINISH, SetVdsStatusVDSCommand, log id: 1319d341 >2018-05-29 18:30:03,505-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6637cf84, Job ID: 586aea81-579f-45c6-8d67-910ae4f20a1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:30:03,510-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36c8bca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:30:03,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6637cf84] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36c8bca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:30:03,522-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7af1a33b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:30:03,523-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7af1a33b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [725abf5] Autorecovering 2 hosts >2018-05-29 18:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [725abf5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:35:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [766f35f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 337b5125 >2018-05-29 18:35:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [766f35f0] FINISH, SetVdsStatusVDSCommand, log id: 337b5125 >2018-05-29 18:35:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [766f35f0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:35:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [766f35f0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:35:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5de07a08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 364622fa >2018-05-29 18:35:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5de07a08] FINISH, SetVdsStatusVDSCommand, log id: 364622fa >2018-05-29 18:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5de07a08] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5247dc61 >2018-05-29 18:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5de07a08] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5247dc61 >2018-05-29 18:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:35:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5de07a08] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:35:01,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [632ea97f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20950321 >2018-05-29 18:35:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [632ea97f] FINISH, GetHardwareInfoVDSCommand, log id: 20950321 >2018-05-29 18:35:01,185-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [64b89242] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:35:01,190-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5fcf7c4b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:35:01,195-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [309ca44e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:35:01,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [309ca44e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4731052d >2018-05-29 18:35:02,020-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fd7cedb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18cb7c08 >2018-05-29 18:35:02,022-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1fd7cedb] FINISH, GetHardwareInfoVDSCommand, log id: 18cb7c08 >2018-05-29 18:35:02,034-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [578e31f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:35:02,039-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7939dec4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:35:02,044-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [55dc5b23] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:35:02,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [55dc5b23] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c3a8b68 >2018-05-29 18:35:02,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:35:02,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [55dc5b23] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c3a8b68 >2018-05-29 18:35:02,437-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,442-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,447-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [55dc5b23] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,449-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [55dc5b23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:35:02,452-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [55dc5b23] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 55dc5b23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:35:02,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3bf419cd >2018-05-29 18:35:02,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:35:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:35:02,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:35:02,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:35:02,557-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:35:02,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3bf419cd >2018-05-29 18:35:02,557-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:35:02,570-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [62bb8445] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:35:02,572-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [62bb8445] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656294f8 >2018-05-29 18:35:02,574-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [62bb8445] FINISH, SetVdsStatusVDSCommand, log id: 656294f8 >2018-05-29 18:35:02,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:35:02,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [309ca44e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4731052d >2018-05-29 18:35:02,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,580-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:35:02,589-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [309ca44e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:35:02,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62bb8445, Job ID: 6a057ab9-cab3-43f2-9182-5d24c37e1786, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:35:02,613-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7939dec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:35:02,615-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [62bb8445] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7939dec4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:35:02,627-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [35b2c9ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:35:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [35b2c9ba] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:35:02,639-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [309ca44e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:35:02,642-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [309ca44e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 309ca44e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:35:02,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2fccd5 >2018-05-29 18:35:02,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:35:02,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:35:02,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:35:02,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2fccd5 >2018-05-29 18:35:02,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:35:02,758-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [c847b4a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:35:02,760-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c847b4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45aea58a >2018-05-29 18:35:02,762-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c847b4a] FINISH, SetVdsStatusVDSCommand, log id: 45aea58a >2018-05-29 18:35:02,770-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: c847b4a, Job ID: a100bf56-739a-4aec-9f46-293ca1cfa504, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:35:02,775-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5fcf7c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:35:02,777-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c847b4a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5fcf7c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:35:02,787-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7cf4e3dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:35:02,789-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7cf4e3dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:40:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [4607c14d] Autorecovering 2 hosts >2018-05-29 18:40:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [4607c14d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [23441e5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c98e7af >2018-05-29 18:40:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [23441e5c] FINISH, SetVdsStatusVDSCommand, log id: 3c98e7af >2018-05-29 18:40:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [23441e5c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:40:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [23441e5c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:40:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:40:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9a2b9f >2018-05-29 18:40:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] FINISH, SetVdsStatusVDSCommand, log id: 9a2b9f >2018-05-29 18:40:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51c5fad4 >2018-05-29 18:40:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [53dc5c1f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51c5fad4 >2018-05-29 18:40:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:40:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [53dc5c1f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:40:00,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [f3fd598] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1edcdcee >2018-05-29 18:40:00,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [f3fd598] FINISH, GetHardwareInfoVDSCommand, log id: 1edcdcee >2018-05-29 18:40:00,267-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4cdadcc2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:40:00,276-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [229412d8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:40:00,281-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f071070] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:40:00,342-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f071070] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1df393c1 >2018-05-29 18:40:00,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3796200e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 122e6d6a >2018-05-29 18:40:00,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3796200e] FINISH, GetHardwareInfoVDSCommand, log id: 122e6d6a >2018-05-29 18:40:00,440-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [989b06a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:40:00,444-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [242870a1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:40:00,449-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [341e0b13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:40:00,455-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [341e0b13] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: de1c74 >2018-05-29 18:40:00,724-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:40:00,724-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4f071070] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1df393c1 >2018-05-29 18:40:00,728-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,728-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,733-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,738-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4f071070] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,740-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4f071070] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:40:00,743-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4f071070] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f071070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:40:00,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5c816e2b >2018-05-29 18:40:00,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:40:00,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:40:00,819-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [341e0b13] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: de1c74 >2018-05-29 18:40:00,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,832-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:40:00,832-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [341e0b13] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:40:00,835-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [341e0b13] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:40:00,837-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [341e0b13] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 341e0b13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:40:00,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:40:00,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:40:00,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5c816e2b >2018-05-29 18:40:00,845-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:40:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ecb9a4e >2018-05-29 18:40:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:40:00,858-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7fdc33e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:40:00,859-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7fdc33e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ab2fdbb >2018-05-29 18:40:00,861-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7fdc33e] FINISH, SetVdsStatusVDSCommand, log id: 7ab2fdbb >2018-05-29 18:40:00,893-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fdc33e, Job ID: 454736f6-27c7-49c5-84e0-f86f80d8304d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:40:00,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 229412d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:40:00,900-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7fdc33e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 229412d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:40:00,911-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [38d45046] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:40:00,913-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [38d45046] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:40:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:40:00,947-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ecb9a4e >2018-05-29 18:40:00,947-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:40:00,959-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [216b55f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:40:00,960-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [216b55f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e3ebbf1 >2018-05-29 18:40:00,962-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [216b55f] FINISH, SetVdsStatusVDSCommand, log id: 6e3ebbf1 >2018-05-29 18:40:00,968-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 216b55f, Job ID: a3281513-6e09-4ed5-b248-0801b22557ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:40:00,972-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 242870a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:40:00,974-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [216b55f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 242870a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:40:00,985-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5fc30682] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:40:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5fc30682] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:45:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [77ac09b0] Autorecovering 2 hosts >2018-05-29 18:45:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [77ac09b0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:45:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [42d466e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 318da4a6 >2018-05-29 18:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [42d466e] FINISH, SetVdsStatusVDSCommand, log id: 318da4a6 >2018-05-29 18:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [42d466e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42d466e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [491a10af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30723bfc >2018-05-29 18:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [491a10af] FINISH, SetVdsStatusVDSCommand, log id: 30723bfc >2018-05-29 18:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [491a10af] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 756177bf >2018-05-29 18:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [491a10af] FINISH, SetHaMaintenanceModeVDSCommand, log id: 756177bf >2018-05-29 18:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:45:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [491a10af] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:45:01,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2e0418f5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e2be2b1 >2018-05-29 18:45:01,514-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [2e0418f5] FINISH, GetHardwareInfoVDSCommand, log id: 3e2be2b1 >2018-05-29 18:45:01,527-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2a028eae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:45:01,535-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4cf6b3a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:45:01,540-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3bf271af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:45:01,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3bf271af] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59ba247a >2018-05-29 18:45:01,611-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [57a3dbf2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e75d441 >2018-05-29 18:45:01,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [57a3dbf2] FINISH, GetHardwareInfoVDSCommand, log id: 7e75d441 >2018-05-29 18:45:01,625-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [6291b814] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:45:01,630-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3925c7f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:45:01,635-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ca58e03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:45:01,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ca58e03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5507934a >2018-05-29 18:45:01,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:45:01,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3bf271af] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59ba247a >2018-05-29 18:45:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:01,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:01,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:01,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:01,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3bf271af] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:01,951-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3bf271af] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:45:01,953-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3bf271af] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bf271af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:45:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2fafae18 >2018-05-29 18:45:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:45:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:45:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:45:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2fafae18 >2018-05-29 18:45:02,057-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:45:02,070-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1655bb90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:45:02,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1655bb90] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1323d7ef >2018-05-29 18:45:02,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1655bb90] FINISH, SetVdsStatusVDSCommand, log id: 1323d7ef >2018-05-29 18:45:02,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1655bb90, Job ID: 39deaf46-4139-4080-9cfb-817326a77930, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:45:02,110-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4cf6b3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:45:02,112-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1655bb90] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4cf6b3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:45:02,123-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3be14d22] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:45:02,124-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3be14d22] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:45:03,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:45:03,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [ca58e03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5507934a >2018-05-29 18:45:03,012-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:03,013-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:03,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:03,017-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:03,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:45:03,022-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [ca58e03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:45:03,024-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [ca58e03] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:45:03,027-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ca58e03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: ca58e03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:45:03,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3748253f >2018-05-29 18:45:03,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:45:03,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:45:03,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:45:03,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3748253f >2018-05-29 18:45:03,127-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:45:03,140-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [4c99d88d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:45:03,141-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4c99d88d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 462565f0 >2018-05-29 18:45:03,144-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4c99d88d] FINISH, SetVdsStatusVDSCommand, log id: 462565f0 >2018-05-29 18:45:03,149-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c99d88d, Job ID: 4618a1e6-289c-45c4-af61-a674a0c58082, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:45:03,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3925c7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:45:03,156-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c99d88d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3925c7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:45:03,166-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [448ae2cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:45:03,167-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [448ae2cb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a3e0424] Autorecovering 2 hosts >2018-05-29 18:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4a3e0424] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4b9fba0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12652ff5 >2018-05-29 18:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4b9fba0f] FINISH, SetVdsStatusVDSCommand, log id: 12652ff5 >2018-05-29 18:50:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4b9fba0f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:50:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [4b9fba0f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [24223fbe] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d516907 >2018-05-29 18:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [24223fbe] FINISH, SetVdsStatusVDSCommand, log id: 2d516907 >2018-05-29 18:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [24223fbe] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 623001c8 >2018-05-29 18:50:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [24223fbe] FINISH, SetHaMaintenanceModeVDSCommand, log id: 623001c8 >2018-05-29 18:50:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [24223fbe] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:50:00,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4dab02fb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: de879be >2018-05-29 18:50:00,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4dab02fb] FINISH, GetHardwareInfoVDSCommand, log id: de879be >2018-05-29 18:50:00,872-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [69982c25] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:50:00,880-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5885c1fb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:50:00,886-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5759b958] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:50:00,897-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5759b958] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73ecb40d >2018-05-29 18:50:01,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:50:01,271-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5759b958] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73ecb40d >2018-05-29 18:50:01,275-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:01,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:01,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:01,281-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:01,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:01,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5759b958] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:01,288-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5759b958] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:50:01,290-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5759b958] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5759b958, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:50:01,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1489dcca >2018-05-29 18:50:01,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:50:01,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:50:01,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:50:01,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1489dcca >2018-05-29 18:50:01,392-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:50:01,405-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4c625bac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:50:01,406-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4c625bac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 225bb82f >2018-05-29 18:50:01,408-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4c625bac] FINISH, SetVdsStatusVDSCommand, log id: 225bb82f >2018-05-29 18:50:01,439-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c625bac, Job ID: a123d66c-e130-475c-9b52-8c21aa3094f8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:50:01,444-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5885c1fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:50:01,446-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4c625bac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5885c1fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:50:01,457-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [df22292] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:50:01,458-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [df22292] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:50:02,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b5bb008] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8c1512e >2018-05-29 18:50:02,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b5bb008] FINISH, GetHardwareInfoVDSCommand, log id: 8c1512e >2018-05-29 18:50:02,769-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [74a7fef1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:50:02,774-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6d405a42] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:50:02,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ac94ec2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:50:02,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ac94ec2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 720adcaf >2018-05-29 18:50:03,159-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:50:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ac94ec2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 720adcaf >2018-05-29 18:50:03,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:03,164-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:03,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:03,168-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:03,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:50:03,173-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ac94ec2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:50:03,176-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ac94ec2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:50:03,178-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ac94ec2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ac94ec2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:50:03,181-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 193c0618 >2018-05-29 18:50:03,181-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:50:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:50:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:50:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 193c0618 >2018-05-29 18:50:03,280-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:50:03,293-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2e49e900] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:50:03,295-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e49e900] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39529b90 >2018-05-29 18:50:03,297-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2e49e900] FINISH, SetVdsStatusVDSCommand, log id: 39529b90 >2018-05-29 18:50:03,302-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e49e900, Job ID: 6bf5e8c9-b488-40c9-8a1a-aa4ef6f91164, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:50:03,307-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d405a42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:50:03,309-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e49e900] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d405a42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:50:03,319-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [38b804ee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:50:03,321-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [38b804ee] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:54:43,200-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [731428f7] Attempting to update VMs/Templates Ovf. >2018-05-29 18:54:43,202-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 18:54:43,202-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 18:54:43,203-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:54:43,207-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 18:54:43,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 18:54:43,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 18:54:43,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 18:54:43,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 18:54:43,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 18:54:43,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [4ad1bba0] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 18:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [56a8143] Autorecovering 2 hosts >2018-05-29 18:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [56a8143] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6d1c2cce] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 477fce0f >2018-05-29 18:55:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6d1c2cce] FINISH, SetVdsStatusVDSCommand, log id: 477fce0f >2018-05-29 18:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6d1c2cce] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 18:55:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6d1c2cce] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 18:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5263d57b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 478d31de >2018-05-29 18:55:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5263d57b] FINISH, SetVdsStatusVDSCommand, log id: 478d31de >2018-05-29 18:55:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5263d57b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 13107763 >2018-05-29 18:55:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5263d57b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 13107763 >2018-05-29 18:55:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 18:55:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5263d57b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 18:55:00,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1009449a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 171c3eff >2018-05-29 18:55:00,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1009449a] FINISH, GetHardwareInfoVDSCommand, log id: 171c3eff >2018-05-29 18:55:00,986-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7a7aab59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:55:00,995-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [ce8300f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:55:01,000-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4c41d4dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:55:01,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4c41d4dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a9c7cb7 >2018-05-29 18:55:01,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:55:01,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4c41d4dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a9c7cb7 >2018-05-29 18:55:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:01,396-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:01,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:01,401-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:01,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:01,405-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4c41d4dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:01,408-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4c41d4dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:55:01,411-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4c41d4dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c41d4dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:55:01,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a2d7634 >2018-05-29 18:55:01,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:55:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:55:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:55:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a2d7634 >2018-05-29 18:55:01,513-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:55:01,527-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [43f4e3d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:55:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43f4e3d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57da54f1 >2018-05-29 18:55:01,531-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43f4e3d4] FINISH, SetVdsStatusVDSCommand, log id: 57da54f1 >2018-05-29 18:55:01,562-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43f4e3d4, Job ID: 8eb7aa9d-256d-463f-96f5-8768ba183075, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:55:01,567-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: ce8300f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:55:01,569-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43f4e3d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: ce8300f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:55:01,580-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [30ef83fb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 18:55:01,581-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [30ef83fb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 18:55:02,165-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [30d1b098] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5e41e3f0 >2018-05-29 18:55:02,167-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [30d1b098] FINISH, GetHardwareInfoVDSCommand, log id: 5e41e3f0 >2018-05-29 18:55:02,180-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7573b7c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:55:02,184-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [35424b07] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:55:02,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [37ed4f66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 18:55:02,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [37ed4f66] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 265a6954 >2018-05-29 18:55:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 18:55:03,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [37ed4f66] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 265a6954 >2018-05-29 18:55:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:03,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:03,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:03,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 18:55:03,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [37ed4f66] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 18:55:03,584-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [37ed4f66] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 18:55:03,587-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37ed4f66] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37ed4f66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 18:55:03,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 40ffdcb4 >2018-05-29 18:55:03,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:55:03,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 18:55:03,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 18:55:03,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 40ffdcb4 >2018-05-29 18:55:03,686-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 18:55:03,700-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [400ac80e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:55:03,701-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [400ac80e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27535a94 >2018-05-29 18:55:03,704-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [400ac80e] FINISH, SetVdsStatusVDSCommand, log id: 27535a94 >2018-05-29 18:55:03,709-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 400ac80e, Job ID: 1b99ae7b-a248-4241-b0e0-75756d10ac78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 18:55:03,714-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35424b07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 18:55:03,716-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [400ac80e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35424b07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 18:55:03,727-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5633c40c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 18:55:03,728-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5633c40c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [606f6b78] Autorecovering 2 hosts >2018-05-29 19:00:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [606f6b78] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c5b1cde] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63d8a64a >2018-05-29 19:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c5b1cde] FINISH, SetVdsStatusVDSCommand, log id: 63d8a64a >2018-05-29 19:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c5b1cde] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6c5b1cde] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [532b002d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7649cc20 >2018-05-29 19:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [532b002d] FINISH, SetVdsStatusVDSCommand, log id: 7649cc20 >2018-05-29 19:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [532b002d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51a60e16 >2018-05-29 19:00:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [532b002d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51a60e16 >2018-05-29 19:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:00:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [532b002d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:00:01,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [15bc5fdc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a67283d >2018-05-29 19:00:01,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [15bc5fdc] FINISH, GetHardwareInfoVDSCommand, log id: 1a67283d >2018-05-29 19:00:01,417-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [3fbc6f94] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:00:01,427-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4b92ceec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:00:01,432-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [9ed7ce8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:00:01,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [9ed7ce8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f2a5ec0 >2018-05-29 19:00:01,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:00:01,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [9ed7ce8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f2a5ec0 >2018-05-29 19:00:01,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:01,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:01,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:01,882-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:01,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:01,887-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [9ed7ce8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:01,890-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [9ed7ce8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:00:01,893-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9ed7ce8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9ed7ce8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:00:01,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 196e078 >2018-05-29 19:00:01,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:00:01,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:00:01,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:00:01,997-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 196e078 >2018-05-29 19:00:01,997-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:00:02,011-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [763fed75] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:00:02,012-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [763fed75] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10f20ec9 >2018-05-29 19:00:02,015-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [763fed75] FINISH, SetVdsStatusVDSCommand, log id: 10f20ec9 >2018-05-29 19:00:02,048-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 763fed75, Job ID: e4201cea-d9c0-4992-a082-7ce70732b3f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:00:02,052-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b92ceec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:00:02,055-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [763fed75] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b92ceec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:00:02,066-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [24fb2211] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:00:02,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [24fb2211] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:00:02,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [29bd49c0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 718c6006 >2018-05-29 19:00:02,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [29bd49c0] FINISH, GetHardwareInfoVDSCommand, log id: 718c6006 >2018-05-29 19:00:02,217-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5caa9684] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:00:02,221-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [625c6dd6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:00:02,226-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2e313e78] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:00:02,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2e313e78] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 94f3c72 >2018-05-29 19:00:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:00:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2e313e78] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 94f3c72 >2018-05-29 19:00:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:02,619-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:02,619-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:02,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:00:02,624-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2e313e78] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:00:02,626-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2e313e78] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:00:02,629-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2e313e78] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2e313e78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:00:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d036d92 >2018-05-29 19:00:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:00:02,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:00:02,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:00:02,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d036d92 >2018-05-29 19:00:02,729-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:00:02,743-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [39fa1264] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:00:02,744-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39fa1264] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 366921fa >2018-05-29 19:00:02,746-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [39fa1264] FINISH, SetVdsStatusVDSCommand, log id: 366921fa >2018-05-29 19:00:02,752-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39fa1264, Job ID: 09d4cf5b-45c8-4af2-a6e5-8952e8e03c25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:00:02,757-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 625c6dd6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:00:02,760-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [39fa1264] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 625c6dd6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:00:02,771-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7460f2aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:00:02,772-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7460f2aa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [38fc79f1] Autorecovering 2 hosts >2018-05-29 19:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [38fc79f1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:05:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2859faf0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18613f68 >2018-05-29 19:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2859faf0] FINISH, SetVdsStatusVDSCommand, log id: 18613f68 >2018-05-29 19:05:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2859faf0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:05:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2859faf0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:05:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49c839fe >2018-05-29 19:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] FINISH, SetVdsStatusVDSCommand, log id: 49c839fe >2018-05-29 19:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 35f569e3 >2018-05-29 19:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [647b9a8d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 35f569e3 >2018-05-29 19:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [647b9a8d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:05:00,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [53f816f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7e20aec5 >2018-05-29 19:05:00,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [53f816f9] FINISH, GetHardwareInfoVDSCommand, log id: 7e20aec5 >2018-05-29 19:05:00,402-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [499ed0e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:05:00,411-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [21ed543a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:05:00,416-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2076e1cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:05:00,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2076e1cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 456565be >2018-05-29 19:05:00,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:05:00,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [2076e1cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 456565be >2018-05-29 19:05:00,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:00,819-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:00,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:00,824-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:00,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:00,828-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [2076e1cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:00,831-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [2076e1cb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:05:00,834-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2076e1cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2076e1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:05:00,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23c734e9 >2018-05-29 19:05:00,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:05:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:05:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:05:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23c734e9 >2018-05-29 19:05:00,940-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:05:00,954-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [4c65de4c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:05:00,955-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c65de4c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 306642d1 >2018-05-29 19:05:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4c65de4c] FINISH, SetVdsStatusVDSCommand, log id: 306642d1 >2018-05-29 19:05:00,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c65de4c, Job ID: 32e304f1-d438-4b2c-894c-9f561cc499de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:05:00,995-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21ed543a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:05:00,997-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c65de4c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21ed543a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:05:01,008-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [40a5dfd1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:05:01,010-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [40a5dfd1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:05:02,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [685f3b40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3b17f16 >2018-05-29 19:05:02,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [685f3b40] FINISH, GetHardwareInfoVDSCommand, log id: 3b17f16 >2018-05-29 19:05:02,776-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3bef2dce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:05:02,780-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2389d97c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:05:02,786-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3d3136d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:05:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3d3136d2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 37d68d82 >2018-05-29 19:05:03,157-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:05:03,157-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3d3136d2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 37d68d82 >2018-05-29 19:05:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:03,162-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:03,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:03,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:03,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:05:03,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3d3136d2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:05:03,173-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3d3136d2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:05:03,176-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d3136d2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d3136d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:05:03,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: fc9d52 >2018-05-29 19:05:03,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:05:03,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:05:03,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:05:03,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: fc9d52 >2018-05-29 19:05:03,276-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:05:03,290-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [7c2a7214] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:05:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c2a7214] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77ec50df >2018-05-29 19:05:03,293-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7c2a7214] FINISH, SetVdsStatusVDSCommand, log id: 77ec50df >2018-05-29 19:05:03,299-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c2a7214, Job ID: d695143f-c255-414b-9573-0fa8ce213117, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:05:03,304-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2389d97c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:05:03,306-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7c2a7214] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2389d97c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:05:03,317-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [209bc4e3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:05:03,318-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [209bc4e3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [50b91b90] Autorecovering 2 hosts >2018-05-29 19:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [50b91b90] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:10:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29a52bda] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 468cd578 >2018-05-29 19:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29a52bda] FINISH, SetVdsStatusVDSCommand, log id: 468cd578 >2018-05-29 19:10:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29a52bda] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:10:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [29a52bda] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:10:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f0550cb >2018-05-29 19:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] FINISH, SetVdsStatusVDSCommand, log id: 1f0550cb >2018-05-29 19:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 25231cff >2018-05-29 19:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6d623c1a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 25231cff >2018-05-29 19:10:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6d623c1a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:10:01,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [582eee8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: b6b6c31 >2018-05-29 19:10:01,032-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [582eee8] FINISH, GetHardwareInfoVDSCommand, log id: b6b6c31 >2018-05-29 19:10:01,045-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [39787bf6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:10:01,053-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3fa8df25] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:10:01,059-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7d86eea1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:10:01,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7d86eea1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2fbfca6f >2018-05-29 19:10:01,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:10:01,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7d86eea1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2fbfca6f >2018-05-29 19:10:01,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:01,444-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:01,448-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:01,448-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:01,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:01,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7d86eea1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:01,457-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7d86eea1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:10:01,459-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d86eea1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d86eea1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:10:01,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b1f42d1 >2018-05-29 19:10:01,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:10:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:10:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:10:01,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b1f42d1 >2018-05-29 19:10:01,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:10:01,574-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [32bb3c3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:10:01,576-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [32bb3c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f713635 >2018-05-29 19:10:01,578-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [32bb3c3] FINISH, SetVdsStatusVDSCommand, log id: 7f713635 >2018-05-29 19:10:01,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6d8e4cf8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23ada0ac >2018-05-29 19:10:01,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6d8e4cf8] FINISH, GetHardwareInfoVDSCommand, log id: 23ada0ac >2018-05-29 19:10:01,598-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [17931338] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:10:01,607-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [17771cf9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:10:01,610-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32bb3c3, Job ID: b556be66-5056-4e56-9411-1f2167735366, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:10:01,612-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7afbb651] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:10:01,615-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3fa8df25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:10:01,617-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [32bb3c3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3fa8df25, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:10:01,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7afbb651] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 463d1b37 >2018-05-29 19:10:01,628-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [71db734c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:10:01,629-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [71db734c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:10:02,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:10:02,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7afbb651] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 463d1b37 >2018-05-29 19:10:02,006-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:02,007-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:02,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:02,011-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:02,015-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:10:02,016-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7afbb651] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:10:02,019-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7afbb651] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:10:02,022-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7afbb651] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7afbb651, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:10:02,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68810767 >2018-05-29 19:10:02,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:10:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:10:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:10:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68810767 >2018-05-29 19:10:02,119-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:10:02,131-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1e90616a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:10:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e90616a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54f1c9bf >2018-05-29 19:10:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1e90616a] FINISH, SetVdsStatusVDSCommand, log id: 54f1c9bf >2018-05-29 19:10:02,140-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1e90616a, Job ID: 0e9bd74e-ea1c-433e-acd7-7682b0602b6b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:10:02,145-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17771cf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:10:02,147-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1e90616a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17771cf9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:10:02,158-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [78ead16e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:10:02,159-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [78ead16e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:15:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5a8f0ef5] Autorecovering 2 hosts >2018-05-29 19:15:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5a8f0ef5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:15:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3698609c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 496dbbe6 >2018-05-29 19:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3698609c] FINISH, SetVdsStatusVDSCommand, log id: 496dbbe6 >2018-05-29 19:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3698609c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:15:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3698609c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:15:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:15:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:15:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61546a1e >2018-05-29 19:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] FINISH, SetVdsStatusVDSCommand, log id: 61546a1e >2018-05-29 19:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e33a128 >2018-05-29 19:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7af6eff5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e33a128 >2018-05-29 19:15:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7af6eff5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:15:02,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [58128c5b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2dbb0334 >2018-05-29 19:15:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [58128c5b] FINISH, GetHardwareInfoVDSCommand, log id: 2dbb0334 >2018-05-29 19:15:02,388-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5aead7a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:15:02,397-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4fbc6be9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:15:02,402-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21518a59] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:15:02,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21518a59] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72cd1d51 >2018-05-29 19:15:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1c87dfd3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5b959942 >2018-05-29 19:15:02,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [1c87dfd3] FINISH, GetHardwareInfoVDSCommand, log id: 5b959942 >2018-05-29 19:15:02,773-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [6ad8a93b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:15:02,777-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [41cd32b6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:15:02,782-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7117d6c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:15:02,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7117d6c4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a760fcc >2018-05-29 19:15:03,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:15:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7117d6c4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4a760fcc >2018-05-29 19:15:03,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,174-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,178-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,183-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7117d6c4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,185-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7117d6c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:15:03,188-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7117d6c4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7117d6c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:15:03,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 781a4a78 >2018-05-29 19:15:03,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:15:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:15:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:15:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 781a4a78 >2018-05-29 19:15:03,291-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:15:03,305-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [65451069] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:15:03,306-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [65451069] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 518b628b >2018-05-29 19:15:03,308-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [65451069] FINISH, SetVdsStatusVDSCommand, log id: 518b628b >2018-05-29 19:15:03,340-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65451069, Job ID: 2cce2ed7-db44-4bde-a775-dfbccf935333, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:15:03,345-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41cd32b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:15:03,347-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [65451069] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41cd32b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:15:03,358-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [48d0c8a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:15:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [48d0c8a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:15:03,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:15:03,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21518a59] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72cd1d51 >2018-05-29 19:15:03,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,786-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,790-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:15:03,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21518a59] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:15:03,798-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21518a59] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:15:03,801-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21518a59] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21518a59, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:15:03,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29b36b22 >2018-05-29 19:15:03,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:15:03,899-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:15:03,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:15:03,899-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:15:03,900-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:15:03,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29b36b22 >2018-05-29 19:15:03,900-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:15:03,913-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c92283] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:15:03,915-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c92283] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12951b95 >2018-05-29 19:15:03,917-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c92283] FINISH, SetVdsStatusVDSCommand, log id: 12951b95 >2018-05-29 19:15:03,922-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c92283, Job ID: a465a10e-692e-48d6-b91f-f141d06fb899, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:15:03,927-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4fbc6be9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:15:03,929-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c92283] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4fbc6be9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:15:03,939-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4779a7ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:15:03,940-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4779a7ed] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:20:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea53f73] Autorecovering 2 hosts >2018-05-29 19:20:00,011-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea53f73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:20:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:20:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [826cee1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b3d4c03 >2018-05-29 19:20:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [826cee1] FINISH, SetVdsStatusVDSCommand, log id: 1b3d4c03 >2018-05-29 19:20:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [826cee1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:20:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [826cee1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:20:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b1d3adf >2018-05-29 19:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] FINISH, SetVdsStatusVDSCommand, log id: 1b1d3adf >2018-05-29 19:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 34e7f64d >2018-05-29 19:20:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [2d26fdd6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 34e7f64d >2018-05-29 19:20:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:20:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2d26fdd6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:20:01,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3cb95ba6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3ce7a1c >2018-05-29 19:20:01,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3cb95ba6] FINISH, GetHardwareInfoVDSCommand, log id: 3ce7a1c >2018-05-29 19:20:01,071-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [b5f2a2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:20:01,080-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [24c3705b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:20:01,086-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [70ba9239] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:20:01,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [70ba9239] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30b03270 >2018-05-29 19:20:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:20:01,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [70ba9239] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30b03270 >2018-05-29 19:20:01,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,484-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,489-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [70ba9239] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,493-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [70ba9239] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:20:01,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70ba9239] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 70ba9239, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:20:01,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4080a147 >2018-05-29 19:20:01,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:20:01,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [318d0c04] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2daff476 >2018-05-29 19:20:01,565-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [318d0c04] FINISH, GetHardwareInfoVDSCommand, log id: 2daff476 >2018-05-29 19:20:01,577-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [51a5d531] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:20:01,582-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [693071c0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:20:01,587-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [10a50851] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:20:01,593-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [10a50851] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 216c0481 >2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:20:01,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:20:01,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:20:01,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4080a147 >2018-05-29 19:20:01,601-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:20:01,614-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [18cf21dd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:20:01,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [18cf21dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f29f40f >2018-05-29 19:20:01,617-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [18cf21dd] FINISH, SetVdsStatusVDSCommand, log id: 5f29f40f >2018-05-29 19:20:01,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18cf21dd, Job ID: 490af0d1-71ee-47a0-9fac-28591756f23d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:20:01,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24c3705b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:20:01,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [18cf21dd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24c3705b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:20:01,666-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [e7922d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:20:01,667-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [e7922d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:20:01,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:20:01,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [10a50851] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 216c0481 >2018-05-29 19:20:01,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,968-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,973-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:20:01,977-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [10a50851] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:20:01,980-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [10a50851] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:20:01,983-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [10a50851] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 10a50851, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:20:01,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f9564e9 >2018-05-29 19:20:01,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:20:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:20:02,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:20:02,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:20:02,082-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:20:02,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f9564e9 >2018-05-29 19:20:02,082-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:20:02,094-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [22d2a8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:20:02,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22d2a8a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1aab1cdc >2018-05-29 19:20:02,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22d2a8a6] FINISH, SetVdsStatusVDSCommand, log id: 1aab1cdc >2018-05-29 19:20:02,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22d2a8a6, Job ID: 3605d5f3-8666-4975-94c1-e87edc49a94b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:20:02,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 693071c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:20:02,109-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22d2a8a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 693071c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:20:02,119-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7b964f5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:20:02,120-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7b964f5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:25:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [73c0daaf] Autorecovering 2 hosts >2018-05-29 19:25:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [73c0daaf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:25:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:25:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:25:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29695546] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5888efdf >2018-05-29 19:25:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [29695546] FINISH, SetVdsStatusVDSCommand, log id: 5888efdf >2018-05-29 19:25:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [29695546] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:25:00,050-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [29695546] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:25:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5932db8c >2018-05-29 19:25:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] FINISH, SetVdsStatusVDSCommand, log id: 5932db8c >2018-05-29 19:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 30baa6c0 >2018-05-29 19:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [6e0feddc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 30baa6c0 >2018-05-29 19:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6e0feddc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:25:02,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fb17a3f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5fed4270 >2018-05-29 19:25:02,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5fb17a3f] FINISH, GetHardwareInfoVDSCommand, log id: 5fed4270 >2018-05-29 19:25:02,255-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5163404f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:25:02,260-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [537d3261] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:25:02,265-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4f0e63b4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:25:02,277-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4f0e63b4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 18cb99fb >2018-05-29 19:25:02,656-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:25:02,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [4f0e63b4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 18cb99fb >2018-05-29 19:25:02,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:02,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:02,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:02,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:02,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [4f0e63b4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:02,673-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [4f0e63b4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:25:02,676-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f0e63b4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4f0e63b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:25:02,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7a0fe7e4 >2018-05-29 19:25:02,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:25:02,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:25:02,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:25:02,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7a0fe7e4 >2018-05-29 19:25:02,780-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:25:02,793-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5a61b335] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:25:02,794-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a61b335] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ff116c8 >2018-05-29 19:25:02,796-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a61b335] FINISH, SetVdsStatusVDSCommand, log id: 7ff116c8 >2018-05-29 19:25:02,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5a61b335, Job ID: 7870d9da-df49-41d0-ac73-b82be5c8d638, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:25:02,809-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 537d3261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:25:02,813-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2fde64d3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5f7370c2 >2018-05-29 19:25:02,813-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5a61b335] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 537d3261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:25:02,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2fde64d3] FINISH, GetHardwareInfoVDSCommand, log id: 5f7370c2 >2018-05-29 19:25:02,826-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3ef076ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:25:02,827-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3ef076ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:25:02,828-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [27cb2639] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:25:02,832-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [257cbb4e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:25:02,837-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [109a9f5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:25:02,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [109a9f5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 626c78ce >2018-05-29 19:25:04,211-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:25:04,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [109a9f5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 626c78ce >2018-05-29 19:25:04,215-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:04,215-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:04,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:04,220-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:04,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:25:04,225-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [109a9f5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:25:04,229-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [109a9f5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:25:04,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [109a9f5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 109a9f5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:25:04,234-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 491862b7 >2018-05-29 19:25:04,234-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:25:04,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:25:04,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:25:04,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 491862b7 >2018-05-29 19:25:04,332-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:25:04,345-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4b32fe28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:25:04,346-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4b32fe28] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2edff71a >2018-05-29 19:25:04,349-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4b32fe28] FINISH, SetVdsStatusVDSCommand, log id: 2edff71a >2018-05-29 19:25:04,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b32fe28, Job ID: e9f62954-21c1-4ee8-9cef-29a5844e68c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:25:04,359-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 257cbb4e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:25:04,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4b32fe28] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 257cbb4e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:25:04,371-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7e1674ff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:25:04,372-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7e1674ff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [368672dd] Autorecovering 2 hosts >2018-05-29 19:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [368672dd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2405589f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 535f7d40 >2018-05-29 19:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2405589f] FINISH, SetVdsStatusVDSCommand, log id: 535f7d40 >2018-05-29 19:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2405589f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2405589f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20e22f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7abc4c18 >2018-05-29 19:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [20e22f6] FINISH, SetVdsStatusVDSCommand, log id: 7abc4c18 >2018-05-29 19:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [20e22f6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: c1b57b4 >2018-05-29 19:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [20e22f6] FINISH, SetHaMaintenanceModeVDSCommand, log id: c1b57b4 >2018-05-29 19:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [20e22f6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:30:00,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5040bb37] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f1b6ec8 >2018-05-29 19:30:00,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5040bb37] FINISH, GetHardwareInfoVDSCommand, log id: 1f1b6ec8 >2018-05-29 19:30:00,429-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2f6672d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:30:00,437-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [6b1fc73c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:30:00,442-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [18ca1a8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:30:00,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [18ca1a8f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f686f04 >2018-05-29 19:30:00,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:30:00,833-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [18ca1a8f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f686f04 >2018-05-29 19:30:00,837-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:00,837-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:00,841-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:00,841-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:00,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:00,846-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [18ca1a8f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:00,848-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [18ca1a8f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:30:00,851-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ca1a8f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18ca1a8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:30:00,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67d6c82c >2018-05-29 19:30:00,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:30:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:30:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:30:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67d6c82c >2018-05-29 19:30:00,957-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-10) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:30:00,970-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [1381138c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:30:00,971-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1381138c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3917a64 >2018-05-29 19:30:00,973-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1381138c] FINISH, SetVdsStatusVDSCommand, log id: 3917a64 >2018-05-29 19:30:01,004-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1381138c, Job ID: 4e8a1d35-8eb7-459f-8d1e-faca59b1417f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:30:01,009-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b1fc73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:30:01,011-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1381138c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b1fc73c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:30:01,022-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [580e738e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:30:01,023-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [580e738e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:30:02,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8648c3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38b40b34 >2018-05-29 19:30:02,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8648c3] FINISH, GetHardwareInfoVDSCommand, log id: 38b40b34 >2018-05-29 19:30:02,065-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6b88bea6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:30:02,069-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [621c81ea] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:30:02,074-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [61b87e8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:30:02,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [61b87e8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 33c910e7 >2018-05-29 19:30:02,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:30:02,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [61b87e8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 33c910e7 >2018-05-29 19:30:02,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:02,453-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:02,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:02,458-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:02,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:30:02,462-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [61b87e8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:30:02,465-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [61b87e8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:30:02,467-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [61b87e8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 61b87e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:30:02,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 412b719d >2018-05-29 19:30:02,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:30:02,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:30:02,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:30:02,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 412b719d >2018-05-29 19:30:02,567-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:30:02,580-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bdf0df4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:30:02,581-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bdf0df4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45f26b87 >2018-05-29 19:30:02,583-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bdf0df4] FINISH, SetVdsStatusVDSCommand, log id: 45f26b87 >2018-05-29 19:30:02,589-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bdf0df4, Job ID: 92ad1f1f-e945-466d-9818-38e1dc4fb2d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:30:02,593-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 621c81ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:30:02,595-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bdf0df4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 621c81ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:30:02,606-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4f5c866] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:30:02,607-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4f5c866] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [93e2fe4] Autorecovering 2 hosts >2018-05-29 19:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [93e2fe4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:35:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:35:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:35:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [c08c658] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 124b159c >2018-05-29 19:35:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [c08c658] FINISH, SetVdsStatusVDSCommand, log id: 124b159c >2018-05-29 19:35:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [c08c658] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:35:00,052-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [c08c658] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:35:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22885f6f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7da05cda >2018-05-29 19:35:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [22885f6f] FINISH, SetVdsStatusVDSCommand, log id: 7da05cda >2018-05-29 19:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22885f6f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4561c8a4 >2018-05-29 19:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [22885f6f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4561c8a4 >2018-05-29 19:35:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [22885f6f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:35:00,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bd79783] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1e61a668 >2018-05-29 19:35:00,299-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bd79783] FINISH, GetHardwareInfoVDSCommand, log id: 1e61a668 >2018-05-29 19:35:00,312-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [39ab223a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:35:00,317-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [661ef0ab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:35:00,322-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74c7078a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:35:00,329-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74c7078a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34ea7188 >2018-05-29 19:35:00,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:35:00,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74c7078a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34ea7188 >2018-05-29 19:35:00,712-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:00,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:00,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:00,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:00,722-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:00,722-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74c7078a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:00,725-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74c7078a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:35:00,728-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74c7078a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74c7078a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:35:00,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c4d67a2 >2018-05-29 19:35:00,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:35:00,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:35:00,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:35:00,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c4d67a2 >2018-05-29 19:35:00,834-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:35:00,847-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [120e8ef6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:35:00,848-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [120e8ef6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: db6ee97 >2018-05-29 19:35:00,851-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [120e8ef6] FINISH, SetVdsStatusVDSCommand, log id: db6ee97 >2018-05-29 19:35:00,881-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 120e8ef6, Job ID: 735a91b5-f6ab-440a-a68f-caba8bc85a99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:35:00,886-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 661ef0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:35:00,888-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [120e8ef6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 661ef0ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:35:00,899-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6e862d4c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:35:00,900-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6e862d4c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:35:01,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7645d898] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 14399609 >2018-05-29 19:35:01,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7645d898] FINISH, GetHardwareInfoVDSCommand, log id: 14399609 >2018-05-29 19:35:01,677-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [f8bd024] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:35:01,681-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [109ddad2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:35:01,686-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ecda47f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:35:01,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ecda47f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e3e2157 >2018-05-29 19:35:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:35:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2ecda47f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e3e2157 >2018-05-29 19:35:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:02,078-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:02,082-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:02,083-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:02,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:35:02,087-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2ecda47f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:35:02,090-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2ecda47f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:35:02,093-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ecda47f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ecda47f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:35:02,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f8e9b80 >2018-05-29 19:35:02,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:35:02,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:35:02,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:35:02,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f8e9b80 >2018-05-29 19:35:02,192-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:35:02,205-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [ee66b56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:35:02,206-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ee66b56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76209c88 >2018-05-29 19:35:02,208-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [ee66b56] FINISH, SetVdsStatusVDSCommand, log id: 76209c88 >2018-05-29 19:35:02,214-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: ee66b56, Job ID: 4585b07d-48bd-4c1b-9159-950e9c0eb15a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:35:02,218-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 109ddad2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:35:02,221-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [ee66b56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 109ddad2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:35:02,232-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2f633630] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:35:02,233-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2f633630] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [706b57d8] Autorecovering 2 hosts >2018-05-29 19:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [706b57d8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [bd5d5ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69782d42 >2018-05-29 19:40:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [bd5d5ca] FINISH, SetVdsStatusVDSCommand, log id: 69782d42 >2018-05-29 19:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [bd5d5ca] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:40:00,080-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [bd5d5ca] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:40:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [21bbb851] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b8cfaeb >2018-05-29 19:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [21bbb851] FINISH, SetVdsStatusVDSCommand, log id: 3b8cfaeb >2018-05-29 19:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [21bbb851] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f03562c >2018-05-29 19:40:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [21bbb851] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f03562c >2018-05-29 19:40:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:40:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [21bbb851] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:40:01,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5d9970c5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 73316bb8 >2018-05-29 19:40:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5d9970c5] FINISH, GetHardwareInfoVDSCommand, log id: 73316bb8 >2018-05-29 19:40:01,643-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7f36373] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:40:01,651-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3165446a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:40:01,656-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a294c88] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:40:01,667-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a294c88] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6dd32c3 >2018-05-29 19:40:02,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:40:02,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7a294c88] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6dd32c3 >2018-05-29 19:40:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:02,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:02,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:02,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7a294c88] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:02,050-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7a294c88] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:40:02,053-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a294c88] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a294c88, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:40:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 410fd28c >2018-05-29 19:40:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:40:02,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:40:02,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:40:02,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 410fd28c >2018-05-29 19:40:02,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:40:02,169-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2980f495] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:40:02,170-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2980f495] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 269df67c >2018-05-29 19:40:02,172-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2980f495] FINISH, SetVdsStatusVDSCommand, log id: 269df67c >2018-05-29 19:40:02,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2980f495, Job ID: 69c88811-a665-4035-9ece-d7f4d6b8af8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:40:02,207-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3165446a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:40:02,209-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2980f495] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3165446a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:40:02,220-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [c9d64ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:40:02,221-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [c9d64ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:40:03,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4b06b282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 490acd2f >2018-05-29 19:40:03,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4b06b282] FINISH, GetHardwareInfoVDSCommand, log id: 490acd2f >2018-05-29 19:40:03,090-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5c5b9666] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:40:03,094-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [30292922] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:40:03,100-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [313075f5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:40:03,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [313075f5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34cbb8b5 >2018-05-29 19:40:03,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:40:03,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [313075f5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 34cbb8b5 >2018-05-29 19:40:03,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:03,493-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:03,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:03,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:40:03,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [313075f5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:40:03,504-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [313075f5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:40:03,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [313075f5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 313075f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18195307 >2018-05-29 19:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:40:03,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:40:03,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:40:03,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18195307 >2018-05-29 19:40:03,605-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:40:03,618-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7b47c4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:40:03,620-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7b47c4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d42da48 >2018-05-29 19:40:03,622-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7b47c4b] FINISH, SetVdsStatusVDSCommand, log id: 4d42da48 >2018-05-29 19:40:03,627-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7b47c4b, Job ID: 190056cb-55a8-4174-8297-5e30ebc88b12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:40:03,632-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 30292922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:40:03,634-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7b47c4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 30292922, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:40:03,645-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3e042be4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:40:03,646-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3e042be4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [70f6c26] Autorecovering 2 hosts >2018-05-29 19:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [70f6c26] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:45:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b5a6b4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 663a88b9 >2018-05-29 19:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b5a6b4] FINISH, SetVdsStatusVDSCommand, log id: 663a88b9 >2018-05-29 19:45:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6b5a6b4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6b5a6b4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:45:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1330353b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71c589b0 >2018-05-29 19:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1330353b] FINISH, SetVdsStatusVDSCommand, log id: 71c589b0 >2018-05-29 19:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1330353b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e228695 >2018-05-29 19:45:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1330353b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e228695 >2018-05-29 19:45:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:45:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1330353b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:45:01,283-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7c8860f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a5cd2e7 >2018-05-29 19:45:01,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7c8860f7] FINISH, GetHardwareInfoVDSCommand, log id: 2a5cd2e7 >2018-05-29 19:45:01,297-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [508e96f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:45:01,305-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [47d15cc6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:45:01,310-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2ca5b686] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:45:01,322-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2ca5b686] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ccad2ba >2018-05-29 19:45:01,701-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:45:01,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2ca5b686] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ccad2ba >2018-05-29 19:45:01,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:01,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:01,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:01,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:01,714-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2ca5b686] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:01,717-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2ca5b686] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:45:01,720-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2ca5b686] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ca5b686, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:45:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 65d2a824 >2018-05-29 19:45:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:45:01,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:45:01,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:45:01,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 65d2a824 >2018-05-29 19:45:01,825-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:45:01,837-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [219713ce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:45:01,839-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [219713ce] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b4a16bd >2018-05-29 19:45:01,841-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [219713ce] FINISH, SetVdsStatusVDSCommand, log id: 4b4a16bd >2018-05-29 19:45:01,872-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 219713ce, Job ID: 7f3fbacb-60e2-4c40-870b-a3947489ed2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:45:01,876-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 47d15cc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:45:01,878-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [219713ce] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 47d15cc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:45:01,889-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [765f012f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:45:01,890-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [765f012f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:45:02,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7011f47] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6db783a >2018-05-29 19:45:02,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7011f47] FINISH, GetHardwareInfoVDSCommand, log id: 6db783a >2018-05-29 19:45:02,931-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [184c41cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:45:02,936-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [75b66999] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:45:02,941-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7f40d0b3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:45:02,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7f40d0b3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42ddb6a5 >2018-05-29 19:45:04,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:45:04,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [7f40d0b3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42ddb6a5 >2018-05-29 19:45:04,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:04,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:04,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:04,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:04,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:45:04,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [7f40d0b3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:45:04,344-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [7f40d0b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:45:04,347-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7f40d0b3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f40d0b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:45:04,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 34a0b062 >2018-05-29 19:45:04,350-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:45:06,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:45:06,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:45:06,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 34a0b062 >2018-05-29 19:45:06,396-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:45:06,409-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [24162269] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:45:06,411-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24162269] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 533b43ae >2018-05-29 19:45:06,413-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24162269] FINISH, SetVdsStatusVDSCommand, log id: 533b43ae >2018-05-29 19:45:06,445-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24162269, Job ID: 41c30954-2a24-4a1a-80d1-040f176c8f9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:45:06,449-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 75b66999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:45:06,452-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24162269] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 75b66999, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:45:06,462-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [621a08ea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:45:06,463-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [621a08ea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1b70da4a] Autorecovering 2 hosts >2018-05-29 19:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1b70da4a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [695b03e8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c71af79 >2018-05-29 19:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [695b03e8] FINISH, SetVdsStatusVDSCommand, log id: 7c71af79 >2018-05-29 19:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [695b03e8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:50:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [695b03e8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1950beae] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15afcba1 >2018-05-29 19:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1950beae] FINISH, SetVdsStatusVDSCommand, log id: 15afcba1 >2018-05-29 19:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1950beae] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 500ed6cb >2018-05-29 19:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1950beae] FINISH, SetHaMaintenanceModeVDSCommand, log id: 500ed6cb >2018-05-29 19:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1950beae] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:50:01,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [28672917] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6725309d >2018-05-29 19:50:01,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [28672917] FINISH, GetHardwareInfoVDSCommand, log id: 6725309d >2018-05-29 19:50:01,101-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3b8f016a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:50:01,109-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1eb53a63] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:50:01,114-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7d2eb441] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:50:01,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7d2eb441] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57611ca6 >2018-05-29 19:50:02,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6d0d4b38] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 8186499 >2018-05-29 19:50:02,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6d0d4b38] FINISH, GetHardwareInfoVDSCommand, log id: 8186499 >2018-05-29 19:50:02,515-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [866fbfd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:50:02,519-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [68e83ef7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:50:02,524-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [24f340ac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:50:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [24f340ac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 471fd7f >2018-05-29 19:50:02,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:50:02,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [24f340ac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 471fd7f >2018-05-29 19:50:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:02,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:02,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:02,915-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:02,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:02,920-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [24f340ac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:02,922-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [24f340ac] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:50:02,925-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24f340ac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24f340ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:50:02,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3962c9b0 >2018-05-29 19:50:02,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:50:03,029-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:50:03,029-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:50:03,029-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3962c9b0 >2018-05-29 19:50:03,029-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:50:03,042-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [148242ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:50:03,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [148242ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b5114a5 >2018-05-29 19:50:03,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [148242ab] FINISH, SetVdsStatusVDSCommand, log id: 3b5114a5 >2018-05-29 19:50:03,078-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 148242ab, Job ID: a8ead82a-9beb-4afb-95b4-d54cec13efdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:50:03,082-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68e83ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:50:03,084-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [148242ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68e83ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:50:03,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5536dc17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:50:03,095-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5536dc17] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:50:03,477-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:50:03,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7d2eb441] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57611ca6 >2018-05-29 19:50:03,481-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:03,481-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:03,485-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:03,486-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:03,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:50:03,490-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7d2eb441] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:50:03,493-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7d2eb441] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:50:03,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7d2eb441] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7d2eb441, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:50:03,498-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35720c58 >2018-05-29 19:50:03,498-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:50:03,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:50:03,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:50:03,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35720c58 >2018-05-29 19:50:03,596-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:50:03,608-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [267a6248] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:50:03,609-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [267a6248] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12454b8 >2018-05-29 19:50:03,611-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [267a6248] FINISH, SetVdsStatusVDSCommand, log id: 12454b8 >2018-05-29 19:50:03,616-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 267a6248, Job ID: 4d825870-5fa7-46b4-9819-ee94ceeb389f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:50:03,620-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1eb53a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:50:03,622-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [267a6248] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1eb53a63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:50:03,632-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [2a4b250] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:50:03,633-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2a4b250] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:54:38,360-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [bbcd86b] Backup check started. >2018-05-29 19:54:38,373-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bbcd86b] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption. >2018-05-29 19:54:38,373-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler2) [bbcd86b] Backup check completed. >2018-05-29 19:54:43,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [1956c667] Attempting to update VMs/Templates Ovf. >2018-05-29 19:54:43,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 19:54:43,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 19:54:43,215-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:54:43,261-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 19:54:43,265-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 19:54:43,265-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 19:54:43,265-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 19:54:43,265-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 19:54:43,265-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 19:54:43,266-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [1cecdcd4] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 19:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1beabcec] Autorecovering 2 hosts >2018-05-29 19:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1beabcec] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146d95b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 541d522 >2018-05-29 19:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146d95b8] FINISH, SetVdsStatusVDSCommand, log id: 541d522 >2018-05-29 19:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146d95b8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 19:55:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [146d95b8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 19:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61456371 >2018-05-29 19:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] FINISH, SetVdsStatusVDSCommand, log id: 61456371 >2018-05-29 19:55:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 33a2583d >2018-05-29 19:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [74f9d2d6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 33a2583d >2018-05-29 19:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 19:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [74f9d2d6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 19:55:00,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bb41ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 750660f5 >2018-05-29 19:55:00,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [47bb41ac] FINISH, GetHardwareInfoVDSCommand, log id: 750660f5 >2018-05-29 19:55:00,690-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [209d1580] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:55:00,698-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [61a992cc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:55:00,703-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7f8bec9f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:55:00,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7f8bec9f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2e6f2a48 >2018-05-29 19:55:01,101-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:55:01,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7f8bec9f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2e6f2a48 >2018-05-29 19:55:01,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,105-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,109-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,113-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,114-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7f8bec9f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,116-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7f8bec9f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:55:01,119-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f8bec9f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f8bec9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:55:01,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7c0f7dd8 >2018-05-29 19:55:01,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:55:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:55:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:55:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7c0f7dd8 >2018-05-29 19:55:01,224-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:55:01,237-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [5c4a5a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:55:01,238-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5c4a5a5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3cf52768 >2018-05-29 19:55:01,241-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [5c4a5a5] FINISH, SetVdsStatusVDSCommand, log id: 3cf52768 >2018-05-29 19:55:01,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c4a5a5, Job ID: 9e343e0b-2989-4bc9-a947-2385b5c3c789, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:55:01,276-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 61a992cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:55:01,278-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c4a5a5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 61a992cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:55:01,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1e8df2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45ed8dc4 >2018-05-29 19:55:01,290-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1e8df2d] FINISH, GetHardwareInfoVDSCommand, log id: 45ed8dc4 >2018-05-29 19:55:01,291-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [208a7e25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 19:55:01,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [208a7e25] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 19:55:01,303-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [1b52fefa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:55:01,311-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7b954f46] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:55:01,316-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [73a5b513] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 19:55:01,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [73a5b513] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 50113953 >2018-05-29 19:55:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 19:55:01,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [73a5b513] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 50113953 >2018-05-29 19:55:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,712-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 19:55:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [73a5b513] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 19:55:01,719-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [73a5b513] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 19:55:01,722-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [73a5b513] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73a5b513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 19:55:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1c74aeac >2018-05-29 19:55:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:55:01,827-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 19:55:01,827-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 19:55:01,827-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1c74aeac >2018-05-29 19:55:01,827-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 19:55:01,841-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [21b70324] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:55:01,842-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21b70324] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6157b6a4 >2018-05-29 19:55:01,844-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [21b70324] FINISH, SetVdsStatusVDSCommand, log id: 6157b6a4 >2018-05-29 19:55:01,876-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 21b70324, Job ID: 69c91fce-efc8-4edd-95b5-07718058fe6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 19:55:01,880-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b954f46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 19:55:01,883-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [21b70324] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b954f46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 19:55:01,893-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1218b081] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 19:55:01,894-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1218b081] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [23938283] Autorecovering 2 hosts >2018-05-29 20:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [23938283] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:00:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:00:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [536fd64d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 655b16fd >2018-05-29 20:00:00,058-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [536fd64d] FINISH, SetVdsStatusVDSCommand, log id: 655b16fd >2018-05-29 20:00:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [536fd64d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:00:00,062-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [536fd64d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [750a168b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3a971265 >2018-05-29 20:00:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [750a168b] FINISH, SetVdsStatusVDSCommand, log id: 3a971265 >2018-05-29 20:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [750a168b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12c513e >2018-05-29 20:00:00,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [750a168b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12c513e >2018-05-29 20:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:00:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [750a168b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:00:02,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [60599506] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 400e6032 >2018-05-29 20:00:02,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [60599506] FINISH, GetHardwareInfoVDSCommand, log id: 400e6032 >2018-05-29 20:00:02,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [64e2980e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:00:02,238-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [192cc2fb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:00:02,243-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74a4d7d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:00:02,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74a4d7d6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e67ad11 >2018-05-29 20:00:02,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4e8ba6ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d2f64e3 >2018-05-29 20:00:02,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [4e8ba6ad] FINISH, GetHardwareInfoVDSCommand, log id: 4d2f64e3 >2018-05-29 20:00:02,599-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [11ebdd82] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:00:02,603-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7d0377b9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:00:02,608-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5cddcb08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:00:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5cddcb08] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30e096b7 >2018-05-29 20:00:02,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:00:02,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [74a4d7d6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e67ad11 >2018-05-29 20:00:02,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:02,631-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:02,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:02,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:02,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [74a4d7d6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:02,643-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [74a4d7d6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:00:02,645-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74a4d7d6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74a4d7d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:00:02,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ed90a3 >2018-05-29 20:00:02,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:00:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:00:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:00:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ed90a3 >2018-05-29 20:00:02,742-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:00:02,755-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [551ebb2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:00:02,756-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [551ebb2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45aabb70 >2018-05-29 20:00:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [551ebb2e] FINISH, SetVdsStatusVDSCommand, log id: 45aabb70 >2018-05-29 20:00:02,788-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 551ebb2e, Job ID: e3196d3d-8766-4c40-8ae0-0ce5c364fc55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:00:02,793-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 192cc2fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:00:02,795-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [551ebb2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 192cc2fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:00:02,806-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1e9e2cda] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:00:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1e9e2cda] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:00:03,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:00:03,978-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5cddcb08] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30e096b7 >2018-05-29 20:00:03,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:03,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:03,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:03,986-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:03,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:00:03,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [5cddcb08] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:00:03,994-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [5cddcb08] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:00:03,996-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5cddcb08] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cddcb08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:00:03,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 928cb12 >2018-05-29 20:00:03,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:00:04,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:00:04,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:00:04,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:00:04,096-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:00:04,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 928cb12 >2018-05-29 20:00:04,097-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:00:04,109-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2871c67b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:00:04,110-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2871c67b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 288ceff2 >2018-05-29 20:00:04,112-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2871c67b] FINISH, SetVdsStatusVDSCommand, log id: 288ceff2 >2018-05-29 20:00:04,118-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2871c67b, Job ID: a969651f-8428-449d-b40f-fe4aad1fa5a8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:00:04,123-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7d0377b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:00:04,125-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2871c67b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7d0377b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:00:04,135-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [64b5e0fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:00:04,136-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [64b5e0fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6451bf01] Autorecovering 2 hosts >2018-05-29 20:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6451bf01] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:05:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7df165c4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bbd2bcb >2018-05-29 20:05:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7df165c4] FINISH, SetVdsStatusVDSCommand, log id: bbd2bcb >2018-05-29 20:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7df165c4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:05:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df165c4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:05:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13ffbd48 >2018-05-29 20:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] FINISH, SetVdsStatusVDSCommand, log id: 13ffbd48 >2018-05-29 20:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7596a29b >2018-05-29 20:05:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30f6e12a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7596a29b >2018-05-29 20:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:05:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30f6e12a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:05:00,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4b062408] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3672ce9a >2018-05-29 20:05:00,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4b062408] FINISH, GetHardwareInfoVDSCommand, log id: 3672ce9a >2018-05-29 20:05:00,431-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2751f760] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:05:00,440-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [60f016e7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:05:00,445-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [648c9641] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:05:00,456-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [648c9641] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1ab50a99 >2018-05-29 20:05:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:05:00,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [648c9641] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1ab50a99 >2018-05-29 20:05:00,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:00,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:00,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:00,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:00,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:00,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [648c9641] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:00,856-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [648c9641] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:05:00,859-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [648c9641] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 648c9641, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:05:00,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 29948371 >2018-05-29 20:05:00,866-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:05:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:05:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:05:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29948371 >2018-05-29 20:05:00,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:05:00,978-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [13b3c988] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:05:00,979-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [13b3c988] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39742eda >2018-05-29 20:05:00,981-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [13b3c988] FINISH, SetVdsStatusVDSCommand, log id: 39742eda >2018-05-29 20:05:01,012-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 13b3c988, Job ID: 2883793e-5591-4f92-85c0-4b37907d0600, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:05:01,017-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 60f016e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:05:01,019-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [13b3c988] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 60f016e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:05:01,029-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [77fd19c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:05:01,030-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [77fd19c4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:05:01,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6294a8bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 461014e9 >2018-05-29 20:05:01,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6294a8bf] FINISH, GetHardwareInfoVDSCommand, log id: 461014e9 >2018-05-29 20:05:01,810-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2d7a0cf9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:05:01,815-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5af3eeb8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:05:01,819-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62d9ef48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:05:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62d9ef48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3af93a06 >2018-05-29 20:05:02,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:05:02,191-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62d9ef48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3af93a06 >2018-05-29 20:05:02,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:02,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:02,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:02,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:02,204-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:05:02,205-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62d9ef48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:05:02,207-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62d9ef48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:05:02,210-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62d9ef48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62d9ef48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:05:02,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 14db2b1f >2018-05-29 20:05:02,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:05:02,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:05:02,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:05:02,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 14db2b1f >2018-05-29 20:05:02,307-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:05:02,320-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bcd64fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:05:02,321-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bcd64fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c03a6cb >2018-05-29 20:05:02,323-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bcd64fd] FINISH, SetVdsStatusVDSCommand, log id: 4c03a6cb >2018-05-29 20:05:02,329-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bcd64fd, Job ID: 644a1957-7dd3-44b3-b88b-ea078a6556b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:05:02,333-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5af3eeb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:05:02,335-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bcd64fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5af3eeb8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:05:02,345-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [32cf816] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:05:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [32cf816] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [23eada57] Autorecovering 2 hosts >2018-05-29 20:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [23eada57] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a7ff7e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4db9a219 >2018-05-29 20:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6a7ff7e3] FINISH, SetVdsStatusVDSCommand, log id: 4db9a219 >2018-05-29 20:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6a7ff7e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:10:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6a7ff7e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76c9b275] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e8803cc >2018-05-29 20:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [76c9b275] FINISH, SetVdsStatusVDSCommand, log id: 1e8803cc >2018-05-29 20:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76c9b275] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 77787967 >2018-05-29 20:10:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [76c9b275] FINISH, SetHaMaintenanceModeVDSCommand, log id: 77787967 >2018-05-29 20:10:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:10:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [76c9b275] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:10:01,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2670160f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: b18545e >2018-05-29 20:10:01,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2670160f] FINISH, GetHardwareInfoVDSCommand, log id: b18545e >2018-05-29 20:10:01,810-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [40163ec7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:10:01,818-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [448622dd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:10:01,823-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [13ed266c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:10:01,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [13ed266c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5f1b064e >2018-05-29 20:10:02,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:10:02,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [13ed266c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5f1b064e >2018-05-29 20:10:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:02,222-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:02,227-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:02,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [13ed266c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:02,234-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [13ed266c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:10:02,237-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [13ed266c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 13ed266c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:10:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 69fe9b38 >2018-05-29 20:10:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:10:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:10:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:10:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 69fe9b38 >2018-05-29 20:10:02,341-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:10:02,354-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [42ba87c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:10:02,355-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42ba87c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22fdb2e3 >2018-05-29 20:10:02,357-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [42ba87c7] FINISH, SetVdsStatusVDSCommand, log id: 22fdb2e3 >2018-05-29 20:10:02,388-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42ba87c7, Job ID: 69a70be3-ff6f-4303-ad95-17f7b0b3527f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:10:02,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 448622dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:10:02,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [42ba87c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 448622dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:10:02,406-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [436084c5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:10:02,407-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [436084c5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:10:03,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43934a74] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 36ed0985 >2018-05-29 20:10:03,032-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43934a74] FINISH, GetHardwareInfoVDSCommand, log id: 36ed0985 >2018-05-29 20:10:03,045-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3c820193] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:10:03,049-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5bcf7856] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:10:03,054-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d0df2d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:10:03,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d0df2d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16d0f4c >2018-05-29 20:10:03,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:10:03,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [d0df2d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16d0f4c >2018-05-29 20:10:03,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:03,433-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:03,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:03,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:03,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:10:03,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [d0df2d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:10:03,443-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [d0df2d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:10:03,446-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d0df2d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d0df2d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:10:03,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fdf8976 >2018-05-29 20:10:03,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:10:03,547-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:10:03,547-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:10:03,547-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fdf8976 >2018-05-29 20:10:03,547-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:10:03,560-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [74ea1fd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:10:03,561-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [74ea1fd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 622c0557 >2018-05-29 20:10:03,563-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [74ea1fd2] FINISH, SetVdsStatusVDSCommand, log id: 622c0557 >2018-05-29 20:10:03,568-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74ea1fd2, Job ID: 22810f62-180a-4008-9250-8f1f2f3b0788, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:10:03,573-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5bcf7856, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:10:03,575-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74ea1fd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5bcf7856, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:10:03,586-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [61013dfc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:10:03,587-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [61013dfc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:15:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [b3b774a] Autorecovering 2 hosts >2018-05-29 20:15:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [b3b774a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:15:00,027-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:15:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:15:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d792420] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4751c55b >2018-05-29 20:15:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7d792420] FINISH, SetVdsStatusVDSCommand, log id: 4751c55b >2018-05-29 20:15:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7d792420] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:15:00,041-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7d792420] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:15:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:15:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:15:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c92e224 >2018-05-29 20:15:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] FINISH, SetVdsStatusVDSCommand, log id: 7c92e224 >2018-05-29 20:15:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 186e3774 >2018-05-29 20:15:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [17fe2e68] FINISH, SetHaMaintenanceModeVDSCommand, log id: 186e3774 >2018-05-29 20:15:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:15:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [17fe2e68] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:15:01,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4580dfa0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e895253 >2018-05-29 20:15:01,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4580dfa0] FINISH, GetHardwareInfoVDSCommand, log id: 7e895253 >2018-05-29 20:15:01,235-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1c1eecfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:15:01,240-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4792a827] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:15:01,245-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5cd5a52a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:15:01,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5cd5a52a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 51d71992 >2018-05-29 20:15:01,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:15:01,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5cd5a52a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 51d71992 >2018-05-29 20:15:01,628-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:01,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:01,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:01,633-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:01,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:01,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5cd5a52a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:01,641-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5cd5a52a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:15:01,643-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5cd5a52a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5cd5a52a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:15:01,646-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4f4e31c1 >2018-05-29 20:15:01,646-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:15:01,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:15:01,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:15:01,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4f4e31c1 >2018-05-29 20:15:01,745-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:15:01,758-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2b189751] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:15:01,759-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2b189751] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1dc73ef5 >2018-05-29 20:15:01,761-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2b189751] FINISH, SetVdsStatusVDSCommand, log id: 1dc73ef5 >2018-05-29 20:15:01,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b189751, Job ID: 2fe445bc-6a3e-4169-b92a-26da9d7bc4bc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:15:01,772-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4792a827, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:15:01,774-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2b189751] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4792a827, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:15:01,785-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [38936568] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:15:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [38936568] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:15:02,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [10808255] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 719c4b81 >2018-05-29 20:15:02,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [10808255] FINISH, GetHardwareInfoVDSCommand, log id: 719c4b81 >2018-05-29 20:15:02,996-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [384121e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:15:03,000-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2665d96] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:15:03,005-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [463e43ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:15:03,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [463e43ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7fba9616 >2018-05-29 20:15:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:15:03,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [463e43ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7fba9616 >2018-05-29 20:15:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:03,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:15:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [463e43ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:15:03,404-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [463e43ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:15:03,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [463e43ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 463e43ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:15:03,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45c9e74d >2018-05-29 20:15:03,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:15:03,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:15:03,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:15:03,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45c9e74d >2018-05-29 20:15:03,508-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:15:03,521-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [8f93b81] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:15:03,522-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [8f93b81] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48fe4ed8 >2018-05-29 20:15:03,524-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [8f93b81] FINISH, SetVdsStatusVDSCommand, log id: 48fe4ed8 >2018-05-29 20:15:03,530-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8f93b81, Job ID: 7f14dfac-cd26-46cf-a9e0-49a02eb83894, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:15:03,534-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2665d96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:15:03,536-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [8f93b81] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2665d96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:15:03,547-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [34e3a8f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:15:03,548-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [34e3a8f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1fe1afb] Autorecovering 2 hosts >2018-05-29 20:20:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1fe1afb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2345b48c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 91bf2c5 >2018-05-29 20:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2345b48c] FINISH, SetVdsStatusVDSCommand, log id: 91bf2c5 >2018-05-29 20:20:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2345b48c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:20:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2345b48c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:20:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [14b958e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 592911e3 >2018-05-29 20:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [14b958e7] FINISH, SetVdsStatusVDSCommand, log id: 592911e3 >2018-05-29 20:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [14b958e7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b8ddaee >2018-05-29 20:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [14b958e7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b8ddaee >2018-05-29 20:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:20:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [14b958e7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:20:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [38e2b4db] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33c485ae >2018-05-29 20:20:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [38e2b4db] FINISH, GetHardwareInfoVDSCommand, log id: 33c485ae >2018-05-29 20:20:01,187-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [257fae78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:20:01,195-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [5c8acf93] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:20:01,201-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e34bf3c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:20:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e34bf3c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85b51c7 >2018-05-29 20:20:01,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:20:01,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e34bf3c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85b51c7 >2018-05-29 20:20:01,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:01,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:01,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:01,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:01,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:01,602-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e34bf3c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:01,604-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e34bf3c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:20:01,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e34bf3c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e34bf3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:20:01,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62bb3cdc >2018-05-29 20:20:01,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:20:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:20:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:20:01,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62bb3cdc >2018-05-29 20:20:01,711-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:20:01,724-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [22589a4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:20:01,725-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22589a4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28fded56 >2018-05-29 20:20:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [22589a4e] FINISH, SetVdsStatusVDSCommand, log id: 28fded56 >2018-05-29 20:20:01,759-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22589a4e, Job ID: d8eeb5e6-22f2-41d7-b29d-af359c459205, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:20:01,764-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c8acf93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:20:01,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [22589a4e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c8acf93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:20:01,776-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [21f92774] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:20:01,777-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [21f92774] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:20:02,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [157022f2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4419d57c >2018-05-29 20:20:02,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [157022f2] FINISH, GetHardwareInfoVDSCommand, log id: 4419d57c >2018-05-29 20:20:02,486-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3934a822] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:20:02,491-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5ca61db2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:20:02,496-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a3db630] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:20:02,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a3db630] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3a563acc >2018-05-29 20:20:03,870-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:20:03,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5a3db630] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3a563acc >2018-05-29 20:20:03,874-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:03,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:03,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:03,879-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:03,883-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:20:03,884-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5a3db630] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:20:03,887-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5a3db630] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:20:03,890-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5a3db630] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a3db630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:20:03,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 52b8dbfb >2018-05-29 20:20:03,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:20:03,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:20:03,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:20:03,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 52b8dbfb >2018-05-29 20:20:03,990-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:20:04,003-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [9207b4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:20:04,004-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [9207b4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2305101c >2018-05-29 20:20:04,006-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [9207b4d] FINISH, SetVdsStatusVDSCommand, log id: 2305101c >2018-05-29 20:20:04,012-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 9207b4d, Job ID: 622300af-9748-480f-b3ae-533b1bb7a540, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:20:04,016-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ca61db2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:20:04,018-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [9207b4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ca61db2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:20:04,029-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1a0ca76f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:20:04,030-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1a0ca76f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [41f39184] Autorecovering 2 hosts >2018-05-29 20:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [41f39184] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:25:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:25:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:25:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1b7b6339] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f545348 >2018-05-29 20:25:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1b7b6339] FINISH, SetVdsStatusVDSCommand, log id: 1f545348 >2018-05-29 20:25:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [1b7b6339] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:25:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1b7b6339] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:25:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3afd8230] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 503d8dc6 >2018-05-29 20:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3afd8230] FINISH, SetVdsStatusVDSCommand, log id: 503d8dc6 >2018-05-29 20:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3afd8230] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2ede1ddd >2018-05-29 20:25:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [3afd8230] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2ede1ddd >2018-05-29 20:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3afd8230] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:25:01,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5a6e39bf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 44923ea2 >2018-05-29 20:25:01,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5a6e39bf] FINISH, GetHardwareInfoVDSCommand, log id: 44923ea2 >2018-05-29 20:25:01,678-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [647b4476] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:25:01,687-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7cdaa4cc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:25:01,693-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4773c29] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:25:01,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4773c29] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6c08112a >2018-05-29 20:25:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:25:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [4773c29] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6c08112a >2018-05-29 20:25:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [4773c29] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,094-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [4773c29] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:25:02,097-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4773c29] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4773c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7546c712 >2018-05-29 20:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:25:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:25:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:25:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7546c712 >2018-05-29 20:25:02,198-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:25:02,211-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [33a90158] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:25:02,212-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [33a90158] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69a35145 >2018-05-29 20:25:02,214-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [33a90158] FINISH, SetVdsStatusVDSCommand, log id: 69a35145 >2018-05-29 20:25:02,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33a90158, Job ID: ce552a05-bf82-49dd-8657-9a0b791839d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:25:02,248-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7cdaa4cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:25:02,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [33a90158] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7cdaa4cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:25:02,261-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [a1d6446] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:25:02,262-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [a1d6446] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:25:02,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7745d6f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 111c6d94 >2018-05-29 20:25:02,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7745d6f4] FINISH, GetHardwareInfoVDSCommand, log id: 111c6d94 >2018-05-29 20:25:02,412-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4aa8989] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:25:02,421-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1a2e901] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:25:02,426-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [477691b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:25:02,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [477691b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 624d5e8f >2018-05-29 20:25:02,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:25:02,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [477691b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 624d5e8f >2018-05-29 20:25:02,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,809-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,814-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:25:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [477691b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:25:02,821-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [477691b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:25:02,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [477691b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 477691b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:25:02,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3753487c >2018-05-29 20:25:02,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:25:02,920-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:25:02,921-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3753487c >2018-05-29 20:25:02,921-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:25:02,933-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [65ee40f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:25:02,935-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65ee40f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 67fcae6b >2018-05-29 20:25:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65ee40f1] FINISH, SetVdsStatusVDSCommand, log id: 67fcae6b >2018-05-29 20:25:02,943-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65ee40f1, Job ID: 2895d503-4cba-4fac-9a45-f07345d3f1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:25:02,947-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a2e901, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:25:02,949-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65ee40f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a2e901, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:25:02,959-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [36f2d41] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:25:02,960-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [36f2d41] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2af9e0d6] Autorecovering 2 hosts >2018-05-29 20:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [2af9e0d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:30:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [efb5c36] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6101a4c5 >2018-05-29 20:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [efb5c36] FINISH, SetVdsStatusVDSCommand, log id: 6101a4c5 >2018-05-29 20:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [efb5c36] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [efb5c36] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:30:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:30:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76e4f338 >2018-05-29 20:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] FINISH, SetVdsStatusVDSCommand, log id: 76e4f338 >2018-05-29 20:30:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ef72963 >2018-05-29 20:30:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [17b4abe7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ef72963 >2018-05-29 20:30:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:30:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [17b4abe7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:30:00,553-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [405d5f2d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7a93d7fe >2018-05-29 20:30:00,555-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [405d5f2d] FINISH, GetHardwareInfoVDSCommand, log id: 7a93d7fe >2018-05-29 20:30:00,568-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [6d03b85b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:30:00,573-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [15e586d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:30:00,578-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3d968777] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:30:00,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3d968777] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e6170fa >2018-05-29 20:30:00,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:30:00,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3d968777] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e6170fa >2018-05-29 20:30:00,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:00,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:00,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:00,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:00,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:00,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3d968777] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:00,991-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3d968777] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:30:00,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d968777] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d968777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:30:01,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3202a2a0 >2018-05-29 20:30:01,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:30:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:30:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:30:01,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3202a2a0 >2018-05-29 20:30:01,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:30:01,112-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [770eaa99] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:30:01,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [770eaa99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5420d97b >2018-05-29 20:30:01,116-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [770eaa99] FINISH, SetVdsStatusVDSCommand, log id: 5420d97b >2018-05-29 20:30:01,147-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 770eaa99, Job ID: a7f756d0-e6c7-4604-b372-85c7df083017, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:30:01,152-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15e586d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:30:01,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [770eaa99] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15e586d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:30:01,165-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [65c75cbb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:30:01,166-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [65c75cbb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:30:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3eb840a1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 597af2f0 >2018-05-29 20:30:02,860-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3eb840a1] FINISH, GetHardwareInfoVDSCommand, log id: 597af2f0 >2018-05-29 20:30:02,873-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [10bb7ba6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:30:02,878-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1f79c1ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:30:02,883-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75fa290e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:30:02,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75fa290e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24deab5e >2018-05-29 20:30:04,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:30:04,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [75fa290e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24deab5e >2018-05-29 20:30:04,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:04,257-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:04,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:04,261-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:04,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:30:04,265-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [75fa290e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:30:04,268-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [75fa290e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:30:04,271-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [75fa290e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75fa290e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:30:04,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6337d758 >2018-05-29 20:30:04,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:30:04,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:30:04,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:30:04,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6337d758 >2018-05-29 20:30:04,373-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:30:04,386-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [8f6c2aa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:30:04,387-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [8f6c2aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66d841f6 >2018-05-29 20:30:04,389-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [8f6c2aa] FINISH, SetVdsStatusVDSCommand, log id: 66d841f6 >2018-05-29 20:30:04,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 8f6c2aa, Job ID: d2acdbcf-940c-4d36-82cf-3af70398a0e1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:30:04,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f79c1ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:30:04,401-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [8f6c2aa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f79c1ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:30:04,411-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1e3c563] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:30:04,412-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1e3c563] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [719b2a49] Autorecovering 2 hosts >2018-05-29 20:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [719b2a49] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:35:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9f31913] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5738975c >2018-05-29 20:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9f31913] FINISH, SetVdsStatusVDSCommand, log id: 5738975c >2018-05-29 20:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9f31913] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:35:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [9f31913] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:35:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b169133 >2018-05-29 20:35:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] FINISH, SetVdsStatusVDSCommand, log id: 5b169133 >2018-05-29 20:35:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1d10078 >2018-05-29 20:35:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [2a621bc2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1d10078 >2018-05-29 20:35:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:35:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [2a621bc2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:35:01,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59226faa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2f5eafe6 >2018-05-29 20:35:01,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [59226faa] FINISH, GetHardwareInfoVDSCommand, log id: 2f5eafe6 >2018-05-29 20:35:01,955-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2fa5c669] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:35:01,964-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52415143] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:35:01,969-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [a7179dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:35:01,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [a7179dc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fc442e1 >2018-05-29 20:35:02,017-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1def6e68] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1b36b63 >2018-05-29 20:35:02,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1def6e68] FINISH, GetHardwareInfoVDSCommand, log id: 1b36b63 >2018-05-29 20:35:02,031-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7ac92934] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:35:02,036-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7199b880] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:35:02,041-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4fd1ab93] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:35:02,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4fd1ab93] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ca5b1cd >2018-05-29 20:35:02,366-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:35:02,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [a7179dc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fc442e1 >2018-05-29 20:35:02,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,371-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,375-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [a7179dc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,383-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [a7179dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:35:02,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [a7179dc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a7179dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:35:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41a20c9e >2018-05-29 20:35:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:35:02,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:35:02,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [4fd1ab93] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ca5b1cd >2018-05-29 20:35:02,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,427-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:35:02,431-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [4fd1ab93] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:35:02,434-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [4fd1ab93] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:35:02,436-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4fd1ab93] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4fd1ab93, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:35:02,489-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:35:02,489-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:35:02,489-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41a20c9e >2018-05-29 20:35:02,489-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:35:02,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7039f10d >2018-05-29 20:35:02,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:35:02,504-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5297d7b8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:35:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5297d7b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21e4cb22 >2018-05-29 20:35:02,507-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5297d7b8] FINISH, SetVdsStatusVDSCommand, log id: 21e4cb22 >2018-05-29 20:35:02,538-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5297d7b8, Job ID: 8e554deb-8ee1-4186-9c88-2026aa630433, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:35:02,543-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52415143, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:35:02,545-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5297d7b8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52415143, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:35:02,556-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [35a1e8c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:35:02,557-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [35a1e8c3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:35:02,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:35:02,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:35:02,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7039f10d >2018-05-29 20:35:02,591-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:35:02,603-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [49340430] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:35:02,605-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [49340430] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 727a368 >2018-05-29 20:35:02,607-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [49340430] FINISH, SetVdsStatusVDSCommand, log id: 727a368 >2018-05-29 20:35:02,612-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49340430, Job ID: c897426c-8cac-407f-bada-f51014a9d270, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:35:02,616-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7199b880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:35:02,619-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [49340430] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7199b880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:35:02,629-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [6a2163b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:35:02,630-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [6a2163b7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [67613628] Autorecovering 2 hosts >2018-05-29 20:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [67613628] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1966e3b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 597befb6 >2018-05-29 20:40:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1966e3b1] FINISH, SetVdsStatusVDSCommand, log id: 597befb6 >2018-05-29 20:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1966e3b1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:40:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1966e3b1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:40:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fb61436] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28d7d784 >2018-05-29 20:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4fb61436] FINISH, SetVdsStatusVDSCommand, log id: 28d7d784 >2018-05-29 20:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4fb61436] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 50b1ac97 >2018-05-29 20:40:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4fb61436] FINISH, SetHaMaintenanceModeVDSCommand, log id: 50b1ac97 >2018-05-29 20:40:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:40:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4fb61436] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:40:00,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [497f18e2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f79c083 >2018-05-29 20:40:00,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [497f18e2] FINISH, GetHardwareInfoVDSCommand, log id: 1f79c083 >2018-05-29 20:40:00,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1d3239c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:40:00,238-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [211cca15] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:40:00,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4a4e4896] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 8afc064 >2018-05-29 20:40:00,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4a4e4896] FINISH, GetHardwareInfoVDSCommand, log id: 8afc064 >2018-05-29 20:40:00,244-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2d2cd9e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:40:00,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2d2cd9e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b21e393 >2018-05-29 20:40:00,255-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [53ff17c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:40:00,260-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [18ad2692] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:40:00,265-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52c79308] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:40:00,278-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52c79308] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d48ec74 >2018-05-29 20:40:00,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:40:00,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2d2cd9e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b21e393 >2018-05-29 20:40:00,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,641-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:40:00,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [52c79308] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d48ec74 >2018-05-29 20:40:00,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2d2cd9e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,649-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,650-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2d2cd9e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:40:00,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2d2cd9e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d2cd9e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:40:00,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,655-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:40:00,659-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [52c79308] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:40:00,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2daa4d45 >2018-05-29 20:40:00,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:40:00,662-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [52c79308] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:40:00,665-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52c79308] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 52c79308, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:40:00,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:40:00,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:40:00,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2daa4d45 >2018-05-29 20:40:00,755-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:40:00,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64b75b86 >2018-05-29 20:40:00,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:40:00,768-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [57807fac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:40:00,770-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57807fac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27116732 >2018-05-29 20:40:00,772-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [57807fac] FINISH, SetVdsStatusVDSCommand, log id: 27116732 >2018-05-29 20:40:00,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 57807fac, Job ID: a9afe43d-6122-44b5-b2e2-d7be39f54165, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:40:00,809-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 211cca15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:40:00,811-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57807fac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 211cca15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:40:00,822-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [19515bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:40:00,823-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [19515bd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:40:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:40:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:40:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64b75b86 >2018-05-29 20:40:00,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:40:00,867-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4f62f049] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:40:00,869-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f62f049] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 424473d9 >2018-05-29 20:40:00,871-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4f62f049] FINISH, SetVdsStatusVDSCommand, log id: 424473d9 >2018-05-29 20:40:00,876-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f62f049, Job ID: 6decfd96-33d1-4b40-af44-91b6ebb4641f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:40:00,881-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18ad2692, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:40:00,883-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4f62f049] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18ad2692, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:40:00,893-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2a387ba6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:40:00,894-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2a387ba6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c367d93] Autorecovering 2 hosts >2018-05-29 20:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c367d93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7d064d62] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c5ad69c >2018-05-29 20:45:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7d064d62] FINISH, SetVdsStatusVDSCommand, log id: c5ad69c >2018-05-29 20:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7d064d62] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7d064d62] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:45:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [40901261] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e605e2a >2018-05-29 20:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [40901261] FINISH, SetVdsStatusVDSCommand, log id: e605e2a >2018-05-29 20:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [40901261] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ee1eafa >2018-05-29 20:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [40901261] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ee1eafa >2018-05-29 20:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:45:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [40901261] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:45:01,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1a0db928] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 730b7b3 >2018-05-29 20:45:01,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1a0db928] FINISH, GetHardwareInfoVDSCommand, log id: 730b7b3 >2018-05-29 20:45:01,753-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [bbcea95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:45:01,761-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [1f3f7342] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:45:01,767-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6b42a8e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:45:01,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6b42a8e0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 525bb020 >2018-05-29 20:45:01,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [37e65e3d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6b4bce7c >2018-05-29 20:45:01,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [37e65e3d] FINISH, GetHardwareInfoVDSCommand, log id: 6b4bce7c >2018-05-29 20:45:01,833-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [42302273] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:45:01,837-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [71cf6ec0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:45:01,842-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [64888bbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:45:01,849-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [64888bbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1eb8c021 >2018-05-29 20:45:02,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:45:02,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6b42a8e0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 525bb020 >2018-05-29 20:45:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,163-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,167-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,172-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6b42a8e0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,174-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6b42a8e0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:45:02,177-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6b42a8e0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b42a8e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:45:02,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4121745e >2018-05-29 20:45:02,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:45:02,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:45:02,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [64888bbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1eb8c021 >2018-05-29 20:45:02,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:45:02,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [64888bbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:45:02,238-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [64888bbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:45:02,240-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [64888bbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 64888bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:45:02,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:45:02,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:45:02,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4121745e >2018-05-29 20:45:02,280-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:45:02,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a82df60 >2018-05-29 20:45:02,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:45:02,294-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [163931e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:45:02,295-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [163931e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c846668 >2018-05-29 20:45:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [163931e6] FINISH, SetVdsStatusVDSCommand, log id: 7c846668 >2018-05-29 20:45:02,328-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 163931e6, Job ID: 5e37b18b-63c3-42ba-9fc2-77091a8ad185, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:45:02,332-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f3f7342, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:45:02,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [163931e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f3f7342, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:45:02,345-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [6e3b1b06] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:45:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [6e3b1b06] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:45:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:45:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:45:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a82df60 >2018-05-29 20:45:02,383-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:45:02,396-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [50037b06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:45:02,397-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [50037b06] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24dd708c >2018-05-29 20:45:02,400-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [50037b06] FINISH, SetVdsStatusVDSCommand, log id: 24dd708c >2018-05-29 20:45:02,433-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 50037b06, Job ID: ec02d248-e42d-4809-ad5b-37aef69faf83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:45:02,438-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 71cf6ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:45:02,440-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [50037b06] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 71cf6ec0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:45:02,450-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [73905950] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:45:02,451-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [73905950] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a3a692e] Autorecovering 2 hosts >2018-05-29 20:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7a3a692e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:50:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:50:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:50:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56801ecd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33c646dc >2018-05-29 20:50:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56801ecd] FINISH, SetVdsStatusVDSCommand, log id: 33c646dc >2018-05-29 20:50:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [56801ecd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:50:00,053-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [56801ecd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:50:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7457c3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fc33926 >2018-05-29 20:50:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f7457c3] FINISH, SetVdsStatusVDSCommand, log id: 2fc33926 >2018-05-29 20:50:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [f7457c3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: a1cf03e >2018-05-29 20:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [f7457c3] FINISH, SetHaMaintenanceModeVDSCommand, log id: a1cf03e >2018-05-29 20:50:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f7457c3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:50:03,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7c989ecf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5c47ae64 >2018-05-29 20:50:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7c989ecf] FINISH, GetHardwareInfoVDSCommand, log id: 5c47ae64 >2018-05-29 20:50:03,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586f18d6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7579b403 >2018-05-29 20:50:03,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [586f18d6] FINISH, GetHardwareInfoVDSCommand, log id: 7579b403 >2018-05-29 20:50:03,087-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4b0e2bdd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:50:03,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [362e12f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:50:03,098-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3dacae4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:50:03,099-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [59e2c72a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:50:03,103-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [650f9c4b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:50:03,104-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [26e222c5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:50:03,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [650f9c4b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35efa60d >2018-05-29 20:50:03,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [26e222c5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3730a043 >2018-05-29 20:50:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:50:03,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [26e222c5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3730a043 >2018-05-29 20:50:03,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:50:03,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [650f9c4b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35efa60d >2018-05-29 20:50:03,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,510-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,514-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,515-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [26e222c5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,518-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [26e222c5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:50:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:50:03,520-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [650f9c4b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:50:03,524-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [650f9c4b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:50:03,524-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [26e222c5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 26e222c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:50:03,526-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [650f9c4b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 650f9c4b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:50:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fdd4d64 >2018-05-29 20:50:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:50:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:50:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:50:03,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fdd4d64 >2018-05-29 20:50:03,624-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:50:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1623ce03 >2018-05-29 20:50:03,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:50:03,637-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2dfbd79] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:50:03,639-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2dfbd79] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ec61568 >2018-05-29 20:50:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2dfbd79] FINISH, SetVdsStatusVDSCommand, log id: 2ec61568 >2018-05-29 20:50:03,672-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2dfbd79, Job ID: 3b62b387-52fa-4334-8973-4a7e23b3f036, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:50:03,677-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 59e2c72a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:50:03,679-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2dfbd79] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 59e2c72a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:50:03,689-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [75474944] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:50:03,690-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [75474944] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:50:03,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:50:03,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:50:03,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1623ce03 >2018-05-29 20:50:03,727-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:50:03,740-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [103dc3c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:50:03,741-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [103dc3c2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1bb6cb01 >2018-05-29 20:50:03,744-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [103dc3c2] FINISH, SetVdsStatusVDSCommand, log id: 1bb6cb01 >2018-05-29 20:50:03,749-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 103dc3c2, Job ID: c0dc0b1e-bdae-4a60-9ca0-1ef4aea528d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:50:03,754-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3dacae4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:50:03,756-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [103dc3c2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3dacae4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:50:03,766-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [56737be9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:50:03,767-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [56737be9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:54:43,266-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [4cef5a96] Attempting to update VMs/Templates Ovf. >2018-05-29 20:54:43,269-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 20:54:43,269-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 20:54:43,270-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:54:43,317-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 20:54:43,321-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 20:54:43,321-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 20:54:43,321-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 20:54:43,321-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 20:54:43,322-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 20:54:43,322-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [37e21f77] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 20:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [57505fce] Autorecovering 2 hosts >2018-05-29 20:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [57505fce] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [43d49d04] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61a84295 >2018-05-29 20:55:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [43d49d04] FINISH, SetVdsStatusVDSCommand, log id: 61a84295 >2018-05-29 20:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [43d49d04] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 20:55:00,054-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [43d49d04] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:55:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 20:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 711a508f >2018-05-29 20:55:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] FINISH, SetVdsStatusVDSCommand, log id: 711a508f >2018-05-29 20:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23f79dc1 >2018-05-29 20:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1f5d797e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23f79dc1 >2018-05-29 20:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 20:55:00,079-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f5d797e] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 1f5d797e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 20:55:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1f5d797e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 20:55:01,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2ea234e9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2114735a >2018-05-29 20:55:01,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2ea234e9] FINISH, GetHardwareInfoVDSCommand, log id: 2114735a >2018-05-29 20:55:01,369-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [25e1b6bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:55:01,378-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [58cd2626] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:55:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2badf97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3e1c1140 >2018-05-29 20:55:01,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2badf97] FINISH, GetHardwareInfoVDSCommand, log id: 3e1c1140 >2018-05-29 20:55:01,383-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [721c18d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:55:01,395-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7c722009] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:55:01,397-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [721c18d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32f7ab5f >2018-05-29 20:55:01,404-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [441071c5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:55:01,409-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [765fa309] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 20:55:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [765fa309] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72acf2b5 >2018-05-29 20:55:01,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:55:01,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [721c18d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32f7ab5f >2018-05-29 20:55:01,783-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,784-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,788-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 20:55:01,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [765fa309] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72acf2b5 >2018-05-29 20:55:01,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,793-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [721c18d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,794-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,795-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,797-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [721c18d8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:55:01,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,799-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,800-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [721c18d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 721c18d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:55:01,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 20:55:01,804-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [765fa309] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 20:55:01,806-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [765fa309] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 20:55:01,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 16e3919a >2018-05-29 20:55:01,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:55:01,809-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [765fa309] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 765fa309, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:55:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:55:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:55:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 16e3919a >2018-05-29 20:55:01,902-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:55:01,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51e94b1 >2018-05-29 20:55:01,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 20:55:01,916-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [77cf89cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:55:01,917-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [77cf89cb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19a139d3 >2018-05-29 20:55:01,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [77cf89cb] FINISH, SetVdsStatusVDSCommand, log id: 19a139d3 >2018-05-29 20:55:01,928-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 77cf89cb, Job ID: f60237cd-dfbe-475f-acf0-089d9cd34ade, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:55:01,933-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 58cd2626, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:55:01,935-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [77cf89cb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 58cd2626, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:55:01,946-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [94fbd54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 20:55:01,947-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [94fbd54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:55:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 20:55:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 20:55:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51e94b1 >2018-05-29 20:55:02,005-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 20:55:02,018-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [680e9374] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:55:02,019-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [680e9374] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2455a25 >2018-05-29 20:55:02,021-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [680e9374] FINISH, SetVdsStatusVDSCommand, log id: 2455a25 >2018-05-29 20:55:02,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 680e9374, Job ID: ede6032b-f717-4388-81df-20aa5ccb00ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 20:55:02,030-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 441071c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 20:55:02,033-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [680e9374] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 441071c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 20:55:02,043-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2789723f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 20:55:02,044-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2789723f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [48798a2] Autorecovering 2 hosts >2018-05-29 21:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [48798a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7389355f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fb748d >2018-05-29 21:00:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7389355f] FINISH, SetVdsStatusVDSCommand, log id: fb748d >2018-05-29 21:00:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7389355f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:00:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7389355f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:00:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a4f641f >2018-05-29 21:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] FINISH, SetVdsStatusVDSCommand, log id: 5a4f641f >2018-05-29 21:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3f3e1346 >2018-05-29 21:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2681ee1c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3f3e1346 >2018-05-29 21:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:00:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2681ee1c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:00:02,518-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [40806728] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2d36e051 >2018-05-29 21:00:02,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [40806728] FINISH, GetHardwareInfoVDSCommand, log id: 2d36e051 >2018-05-29 21:00:02,533-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [22c576d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:00:02,542-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [5382b75d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:00:02,547-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1b8d4495] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:00:02,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1b8d4495] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 149d937c >2018-05-29 21:00:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6df3c88f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1f4ea2a0 >2018-05-29 21:00:02,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6df3c88f] FINISH, GetHardwareInfoVDSCommand, log id: 1f4ea2a0 >2018-05-29 21:00:02,747-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [676e793e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:00:02,752-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [31d9efeb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:00:02,757-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5c1e1808] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:00:02,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5c1e1808] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61898d8c >2018-05-29 21:00:02,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:00:02,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1b8d4495] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 149d937c >2018-05-29 21:00:02,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:02,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:02,953-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1b8d4495] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:02,956-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1b8d4495] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:00:02,958-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1b8d4495] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1b8d4495, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:00:02,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1efd34e1 >2018-05-29 21:00:02,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:00:03,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:00:03,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:00:03,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1efd34e1 >2018-05-29 21:00:03,061-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:00:03,074-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [1cd3731b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:00:03,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1cd3731b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37c398f >2018-05-29 21:00:03,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1cd3731b] FINISH, SetVdsStatusVDSCommand, log id: 37c398f >2018-05-29 21:00:03,109-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1cd3731b, Job ID: e132b390-fbb5-4de4-8e7d-628bb3076175, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:00:03,114-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5382b75d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:00:03,116-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1cd3731b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5382b75d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:00:03,126-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [68eeb3ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:00:03,128-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [68eeb3ae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:00:03,140-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:00:03,140-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5c1e1808] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61898d8c >2018-05-29 21:00:03,144-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:03,145-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:03,149-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:03,153-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:00:03,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5c1e1808] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:00:03,156-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5c1e1808] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:00:03,159-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c1e1808] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c1e1808, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:00:03,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 551fc5b >2018-05-29 21:00:03,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:00:03,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:00:03,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:00:03,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 551fc5b >2018-05-29 21:00:03,267-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:00:03,280-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [4362b678] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:00:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4362b678] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d42692b >2018-05-29 21:00:03,283-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4362b678] FINISH, SetVdsStatusVDSCommand, log id: 6d42692b >2018-05-29 21:00:03,289-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4362b678, Job ID: ba42b361-a776-4d17-b2ec-e6393dc02109, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:00:03,293-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 31d9efeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:00:03,295-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4362b678] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 31d9efeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:00:03,305-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [24bdb77b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:00:03,307-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [24bdb77b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a5887be] Autorecovering 2 hosts >2018-05-29 21:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7a5887be] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:05:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:05:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:05:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6e5b73af] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 747f333c >2018-05-29 21:05:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6e5b73af] FINISH, SetVdsStatusVDSCommand, log id: 747f333c >2018-05-29 21:05:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,059-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6e5b73af] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 6e5b73af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 21:05:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [6e5b73af] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:05:00,060-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6e5b73af] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:05:00,113-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:05:00,119-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:05:00,119-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,119-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,121-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13627ef4 >2018-05-29 21:05:00,123-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] FINISH, SetVdsStatusVDSCommand, log id: 13627ef4 >2018-05-29 21:05:00,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 68a08001 >2018-05-29 21:05:00,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [2b5d1b18] FINISH, SetHaMaintenanceModeVDSCommand, log id: 68a08001 >2018-05-29 21:05:00,128-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:05:00,130-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2b5d1b18] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:05:00,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3faa2b03] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 466815e0 >2018-05-29 21:05:00,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3faa2b03] FINISH, GetHardwareInfoVDSCommand, log id: 466815e0 >2018-05-29 21:05:00,744-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4c4a2f30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:05:00,753-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4199a911] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:05:00,758-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22c2d365] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:05:00,764-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22c2d365] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6db50aeb >2018-05-29 21:05:00,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6c0a8ead] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 787b362f >2018-05-29 21:05:00,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6c0a8ead] FINISH, GetHardwareInfoVDSCommand, log id: 787b362f >2018-05-29 21:05:00,963-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [21275878] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:05:00,967-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [7f296c61] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:05:00,972-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [351d82a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:05:00,978-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [351d82a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fe923c4 >2018-05-29 21:05:01,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:05:01,152-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [22c2d365] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6db50aeb >2018-05-29 21:05:01,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:01,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:01,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:01,161-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:01,165-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:01,166-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [22c2d365] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:01,168-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [22c2d365] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:05:01,170-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [22c2d365] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 22c2d365, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:05:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 46826498 >2018-05-29 21:05:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:05:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:05:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:05:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 46826498 >2018-05-29 21:05:01,270-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:05:01,283-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [306f7f3e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:05:01,284-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [306f7f3e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62331714 >2018-05-29 21:05:01,286-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [306f7f3e] FINISH, SetVdsStatusVDSCommand, log id: 62331714 >2018-05-29 21:05:01,315-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 306f7f3e, Job ID: b4def737-96ef-4044-8a4d-a3a0c7a3cc74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:05:01,320-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4199a911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:05:01,322-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [306f7f3e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4199a911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:05:01,333-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [679a8468] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:05:01,334-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [679a8468] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:05:02,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:05:02,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [351d82a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fe923c4 >2018-05-29 21:05:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:02,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:02,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:02,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:02,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:05:02,363-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [351d82a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:05:02,366-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [351d82a0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:05:02,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [351d82a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 351d82a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:05:02,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57869e5f >2018-05-29 21:05:02,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:05:02,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:05:02,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:05:02,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57869e5f >2018-05-29 21:05:02,471-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:05:02,484-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [2071067a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:05:02,485-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2071067a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b1f9c5a >2018-05-29 21:05:02,487-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2071067a] FINISH, SetVdsStatusVDSCommand, log id: 1b1f9c5a >2018-05-29 21:05:02,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2071067a, Job ID: 0ff06924-9827-4384-aa66-587e2a12934b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:05:02,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f296c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:05:02,500-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2071067a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f296c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:05:02,511-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [ebc71bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:05:02,512-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [ebc71bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:10:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1bac5d4] Autorecovering 2 hosts >2018-05-29 21:10:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1bac5d4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:10:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:10:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:10:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [631b1ef3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7326792e >2018-05-29 21:10:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [631b1ef3] FINISH, SetVdsStatusVDSCommand, log id: 7326792e >2018-05-29 21:10:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [631b1ef3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:10:00,048-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [631b1ef3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:10:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1edf3bc9 >2018-05-29 21:10:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] FINISH, SetVdsStatusVDSCommand, log id: 1edf3bc9 >2018-05-29 21:10:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 482c4323 >2018-05-29 21:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4bdddea3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 482c4323 >2018-05-29 21:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4bdddea3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:10:00,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [65b59aff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2c6975c6 >2018-05-29 21:10:00,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [65b59aff] FINISH, GetHardwareInfoVDSCommand, log id: 2c6975c6 >2018-05-29 21:10:00,227-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [5eae1697] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:10:00,235-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [79df117c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:10:00,241-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [539fad5c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:10:00,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [539fad5c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6999aa1a >2018-05-29 21:10:00,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:10:00,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [539fad5c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6999aa1a >2018-05-29 21:10:00,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:00,632-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:00,635-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:00,636-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:00,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:00,640-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [539fad5c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:00,643-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [539fad5c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:10:00,645-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [539fad5c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 539fad5c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:10:00,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ca0c4f1 >2018-05-29 21:10:00,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:10:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:10:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:10:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: ca0c4f1 >2018-05-29 21:10:00,753-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:10:00,766-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [74743ba5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:10:00,767-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [74743ba5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c2b36e5 >2018-05-29 21:10:00,769-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [74743ba5] FINISH, SetVdsStatusVDSCommand, log id: 1c2b36e5 >2018-05-29 21:10:00,777-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74743ba5, Job ID: 319e10ff-cd19-411a-b67f-98a6355056cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:10:00,781-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79df117c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:10:00,784-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [74743ba5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79df117c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:10:00,794-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4414b58b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:10:00,795-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4414b58b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:10:01,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5231a638] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4a998fc7 >2018-05-29 21:10:01,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5231a638] FINISH, GetHardwareInfoVDSCommand, log id: 4a998fc7 >2018-05-29 21:10:01,960-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1b8e08ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:10:01,964-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1d78fb20] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:10:01,970-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [262b095] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:10:01,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [262b095] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e11bd2 >2018-05-29 21:10:02,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:10:02,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [262b095] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e11bd2 >2018-05-29 21:10:02,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:02,361-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:02,365-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:02,365-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:02,369-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:10:02,370-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [262b095] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:10:02,373-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [262b095] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:10:02,375-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [262b095] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 262b095, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:10:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7ab004fd >2018-05-29 21:10:02,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:10:02,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:10:02,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:10:02,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7ab004fd >2018-05-29 21:10:02,472-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:10:02,485-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [c66cf12] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:10:02,486-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c66cf12] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6899ca56 >2018-05-29 21:10:02,489-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [c66cf12] FINISH, SetVdsStatusVDSCommand, log id: 6899ca56 >2018-05-29 21:10:02,494-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: c66cf12, Job ID: 3ff620e4-3a9d-489f-9806-c2ba82d73264, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:10:02,499-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d78fb20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:10:02,501-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [c66cf12] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d78fb20, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:10:02,511-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [224aaddd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:10:02,512-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [224aaddd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [381ae06f] Autorecovering 2 hosts >2018-05-29 21:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [381ae06f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:15:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:15:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:15:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [780ce812] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51eb280a >2018-05-29 21:15:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [780ce812] FINISH, SetVdsStatusVDSCommand, log id: 51eb280a >2018-05-29 21:15:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [780ce812] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:15:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [780ce812] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:15:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:15:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:15:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4767cf6 >2018-05-29 21:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] FINISH, SetVdsStatusVDSCommand, log id: 4767cf6 >2018-05-29 21:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 300867f9 >2018-05-29 21:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [6aa0af87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 300867f9 >2018-05-29 21:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [6aa0af87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:15:01,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [37d2b514] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5be3cefc >2018-05-29 21:15:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [37d2b514] FINISH, GetHardwareInfoVDSCommand, log id: 5be3cefc >2018-05-29 21:15:01,518-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [10362b06] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:15:01,526-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4b869451] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:15:01,531-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [489bd871] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:15:01,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [489bd871] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3bc07f7e >2018-05-29 21:15:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:15:01,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [489bd871] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3bc07f7e >2018-05-29 21:15:01,909-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:01,910-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:01,913-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:01,914-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:01,918-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:01,918-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [489bd871] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:01,921-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [489bd871] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:15:01,923-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [489bd871] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 489bd871, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:15:01,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 165d334a >2018-05-29 21:15:01,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:15:02,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:15:02,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:15:02,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 165d334a >2018-05-29 21:15:02,026-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:15:02,039-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [438f02a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:15:02,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [438f02a6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 475528f8 >2018-05-29 21:15:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [438f02a6] FINISH, SetVdsStatusVDSCommand, log id: 475528f8 >2018-05-29 21:15:02,075-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 438f02a6, Job ID: 9e42e485-c1c8-4ceb-a0c3-c95099ebfdc6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:15:02,079-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b869451, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:15:02,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [438f02a6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b869451, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:15:02,092-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59b6d365] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:15:02,093-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59b6d365] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:15:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d7b9103] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 66014035 >2018-05-29 21:15:03,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1d7b9103] FINISH, GetHardwareInfoVDSCommand, log id: 66014035 >2018-05-29 21:15:03,192-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [343dfa6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:15:03,197-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [77035c56] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:15:03,202-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12737317] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:15:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12737317] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65b596b2 >2018-05-29 21:15:03,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:15:03,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [12737317] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65b596b2 >2018-05-29 21:15:03,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:03,588-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:03,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:03,593-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:03,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:15:03,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [12737317] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:15:03,600-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [12737317] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:15:03,602-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [12737317] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 12737317, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:15:03,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215e0aa >2018-05-29 21:15:03,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:15:03,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:15:03,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:15:03,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215e0aa >2018-05-29 21:15:03,702-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:15:03,715-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [4f9b8501] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:15:03,716-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4f9b8501] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50a7bb7a >2018-05-29 21:15:03,718-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [4f9b8501] FINISH, SetVdsStatusVDSCommand, log id: 50a7bb7a >2018-05-29 21:15:03,724-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f9b8501, Job ID: d8e2b6e5-ecd3-4714-862d-3fe148a6ba10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:15:03,729-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77035c56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:15:03,731-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4f9b8501] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77035c56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:15:03,742-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6ce398db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:15:03,743-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6ce398db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:20:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [543eb6ea] Autorecovering 2 hosts >2018-05-29 21:20:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [543eb6ea] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:20:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e2a39c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e90b5f6 >2018-05-29 21:20:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e2a39c1] FINISH, SetVdsStatusVDSCommand, log id: 5e90b5f6 >2018-05-29 21:20:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [6e2a39c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:20:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6e2a39c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:20:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 454a1b8c >2018-05-29 21:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] FINISH, SetVdsStatusVDSCommand, log id: 454a1b8c >2018-05-29 21:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 647ec951 >2018-05-29 21:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7b5bd4b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 647ec951 >2018-05-29 21:20:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:20:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7b5bd4b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:20:01,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1dde2490] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7bb697b2 >2018-05-29 21:20:01,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [1dde2490] FINISH, GetHardwareInfoVDSCommand, log id: 7bb697b2 >2018-05-29 21:20:01,364-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4aa6d213] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:20:01,372-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [271dd3e0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:20:01,377-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [364463d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:20:01,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [364463d2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ef27ca8 >2018-05-29 21:20:01,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:20:01,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [364463d2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ef27ca8 >2018-05-29 21:20:01,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:01,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:01,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:01,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:01,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:01,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [364463d2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:01,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [364463d2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:20:01,782-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [364463d2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 364463d2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:20:01,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4ce5a040 >2018-05-29 21:20:01,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:20:01,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:20:01,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:20:01,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:20:01,885-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:20:01,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4ce5a040 >2018-05-29 21:20:01,885-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:20:01,898-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [3d1acb4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:20:01,899-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3d1acb4f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2cc20b8d >2018-05-29 21:20:01,901-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3d1acb4f] FINISH, SetVdsStatusVDSCommand, log id: 2cc20b8d >2018-05-29 21:20:01,932-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d1acb4f, Job ID: 2e9cb2d4-c695-43dd-bf64-e498da7d8c97, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:20:01,936-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 271dd3e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:20:01,938-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3d1acb4f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 271dd3e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:20:01,948-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [5419ad82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:20:01,949-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5419ad82] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:20:02,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2bc38e4b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 324a8bfe >2018-05-29 21:20:02,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2bc38e4b] FINISH, GetHardwareInfoVDSCommand, log id: 324a8bfe >2018-05-29 21:20:02,957-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6cc600e1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:20:02,962-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [55772243] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:20:02,967-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [39803d00] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:20:02,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [39803d00] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c92f500 >2018-05-29 21:20:03,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:20:03,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [39803d00] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c92f500 >2018-05-29 21:20:03,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:03,355-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:03,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:03,360-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:03,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:20:03,364-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [39803d00] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:20:03,367-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [39803d00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:20:03,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [39803d00] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39803d00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:20:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3970c7bc >2018-05-29 21:20:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:20:03,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:20:03,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:20:03,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3970c7bc >2018-05-29 21:20:03,467-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:20:03,480-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [178e6aea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:20:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [178e6aea] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 411bf957 >2018-05-29 21:20:03,483-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [178e6aea] FINISH, SetVdsStatusVDSCommand, log id: 411bf957 >2018-05-29 21:20:03,488-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 178e6aea, Job ID: 93c31075-3ebd-46c4-b757-e1fb45963eb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:20:03,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55772243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:20:03,495-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [178e6aea] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55772243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:20:03,505-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6db011a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:20:03,506-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6db011a8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [50ea0434] Autorecovering 2 hosts >2018-05-29 21:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [50ea0434] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:25:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:25:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:25:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79c09aa0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 38864050 >2018-05-29 21:25:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [79c09aa0] FINISH, SetVdsStatusVDSCommand, log id: 38864050 >2018-05-29 21:25:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [79c09aa0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:25:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [79c09aa0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:25:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:25:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60cff827] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 409e0580 >2018-05-29 21:25:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60cff827] FINISH, SetVdsStatusVDSCommand, log id: 409e0580 >2018-05-29 21:25:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [60cff827] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2f99513b >2018-05-29 21:25:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [60cff827] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2f99513b >2018-05-29 21:25:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:25:00,106-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [60cff827] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:25:01,137-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7da91647] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 212ccc81 >2018-05-29 21:25:01,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7da91647] FINISH, GetHardwareInfoVDSCommand, log id: 212ccc81 >2018-05-29 21:25:01,153-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [6699b168] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:25:01,161-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [18a02a44] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:25:01,166-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5a8e2d48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:25:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5a8e2d48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2cf90112 >2018-05-29 21:25:02,551-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:25:02,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5a8e2d48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2cf90112 >2018-05-29 21:25:02,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:02,556-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:02,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:02,561-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:02,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:02,566-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5a8e2d48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:02,570-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5a8e2d48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:25:02,574-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5a8e2d48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a8e2d48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:25:02,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 38088366 >2018-05-29 21:25:02,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:25:02,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:25:02,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:25:02,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 38088366 >2018-05-29 21:25:02,680-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:25:02,708-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4e953eb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:25:02,710-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4e953eb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6a1ec433 >2018-05-29 21:25:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4e953eb1] FINISH, SetVdsStatusVDSCommand, log id: 6a1ec433 >2018-05-29 21:25:02,746-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e953eb1, Job ID: 972b4e5e-4d57-48d7-b4b8-de17ce379f3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:25:02,751-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18a02a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:25:02,754-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e953eb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18a02a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:25:02,765-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [532c52b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:25:02,766-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [532c52b9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:25:02,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4cf31885] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f80074f >2018-05-29 21:25:02,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4cf31885] FINISH, GetHardwareInfoVDSCommand, log id: 1f80074f >2018-05-29 21:25:02,857-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [442ec85c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:25:02,865-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [68fccb6e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:25:02,870-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4ebb7b03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:25:02,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4ebb7b03] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6baced66 >2018-05-29 21:25:03,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:25:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4ebb7b03] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6baced66 >2018-05-29 21:25:03,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:03,269-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:03,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:03,273-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:03,278-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:25:03,278-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4ebb7b03] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:25:03,281-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4ebb7b03] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:25:03,284-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4ebb7b03] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4ebb7b03, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:25:03,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ba176c9 >2018-05-29 21:25:03,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:25:03,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:25:03,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:25:03,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: ba176c9 >2018-05-29 21:25:03,382-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:25:03,396-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [325ab866] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:25:03,397-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [325ab866] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d91c7ff >2018-05-29 21:25:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [325ab866] FINISH, SetVdsStatusVDSCommand, log id: 2d91c7ff >2018-05-29 21:25:03,405-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 325ab866, Job ID: 0c390c9d-2d6e-44c8-8706-89e757c61204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:25:03,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 68fccb6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:25:03,412-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [325ab866] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 68fccb6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:25:03,423-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [73cee93d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:25:03,424-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [73cee93d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e7174dd] Autorecovering 2 hosts >2018-05-29 21:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e7174dd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2663b02f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f073be0 >2018-05-29 21:30:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2663b02f] FINISH, SetVdsStatusVDSCommand, log id: 1f073be0 >2018-05-29 21:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2663b02f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:30:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2663b02f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bc6b66a >2018-05-29 21:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] FINISH, SetVdsStatusVDSCommand, log id: 3bc6b66a >2018-05-29 21:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5457e97f >2018-05-29 21:30:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [24f0f6a3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5457e97f >2018-05-29 21:30:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:30:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [24f0f6a3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:30:00,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [541898f0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 22b27601 >2018-05-29 21:30:00,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [541898f0] FINISH, GetHardwareInfoVDSCommand, log id: 22b27601 >2018-05-29 21:30:00,495-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [631d05ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:30:00,503-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70451c85] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:30:00,509-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [530a9a60] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:30:00,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [530a9a60] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61c71ff9 >2018-05-29 21:30:00,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:30:00,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [530a9a60] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61c71ff9 >2018-05-29 21:30:00,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:00,903-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:00,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:00,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [530a9a60] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:00,916-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [530a9a60] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:30:00,918-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [530a9a60] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 530a9a60, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:30:00,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5012276a >2018-05-29 21:30:00,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:30:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:30:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:30:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5012276a >2018-05-29 21:30:01,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:30:01,064-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [23e358af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:30:01,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [761e4b5e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 19703a2e >2018-05-29 21:30:01,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [23e358af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52b81dfc >2018-05-29 21:30:01,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [761e4b5e] FINISH, GetHardwareInfoVDSCommand, log id: 19703a2e >2018-05-29 21:30:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [23e358af] FINISH, SetVdsStatusVDSCommand, log id: 52b81dfc >2018-05-29 21:30:01,081-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [37ea5e9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:30:01,090-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [308fb2f5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:30:01,096-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [490679b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:30:01,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23e358af, Job ID: 54d4746e-9621-42bf-b9bc-c3c5f4eba343, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:30:01,108-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70451c85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:30:01,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [490679b5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d3d7aa6 >2018-05-29 21:30:01,111-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [23e358af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70451c85, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:30:01,122-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [19473fcf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:30:01,123-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [19473fcf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:30:01,488-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:30:01,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [490679b5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d3d7aa6 >2018-05-29 21:30:01,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:01,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:01,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:01,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:01,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:30:01,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [490679b5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:30:01,505-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [490679b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:30:01,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490679b5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 490679b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:30:01,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 12814588 >2018-05-29 21:30:01,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:30:01,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:30:01,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:30:01,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 12814588 >2018-05-29 21:30:01,605-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:30:01,618-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [49de8836] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:30:01,619-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49de8836] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4e18e0fb >2018-05-29 21:30:01,622-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [49de8836] FINISH, SetVdsStatusVDSCommand, log id: 4e18e0fb >2018-05-29 21:30:01,630-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 49de8836, Job ID: 3c192c7a-c4d9-4b5b-92e7-ab6cefdcf115, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:30:01,635-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 308fb2f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:30:01,637-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [49de8836] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 308fb2f5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:30:01,647-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [41bdf60] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:30:01,648-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [41bdf60] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [536afe6d] Autorecovering 2 hosts >2018-05-29 21:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [536afe6d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:35:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5886dae9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ad8035d >2018-05-29 21:35:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5886dae9] FINISH, SetVdsStatusVDSCommand, log id: 3ad8035d >2018-05-29 21:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5886dae9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:35:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5886dae9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:35:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [14f491a0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29e40cf1 >2018-05-29 21:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [14f491a0] FINISH, SetVdsStatusVDSCommand, log id: 29e40cf1 >2018-05-29 21:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [14f491a0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 724b8282 >2018-05-29 21:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [14f491a0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 724b8282 >2018-05-29 21:35:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:35:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [14f491a0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:35:01,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3e02b3cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 580fc5dd >2018-05-29 21:35:01,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [3e02b3cb] FINISH, GetHardwareInfoVDSCommand, log id: 580fc5dd >2018-05-29 21:35:01,887-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [14dc8e2c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:35:01,896-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [5335aa18] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:35:01,901-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4749cca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:35:01,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4749cca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d4ee2ce >2018-05-29 21:35:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f6f956b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 26c81057 >2018-05-29 21:35:02,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f6f956b] FINISH, GetHardwareInfoVDSCommand, log id: 26c81057 >2018-05-29 21:35:02,273-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [27f2c8dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:35:02,277-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5f88fefa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:35:02,283-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [771a90e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:35:02,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [771a90e2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5ed7a800 >2018-05-29 21:35:02,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:35:02,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [771a90e2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5ed7a800 >2018-05-29 21:35:02,678-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:02,679-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:02,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:02,683-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:02,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:02,688-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [771a90e2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:02,691-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [771a90e2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:35:02,693-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [771a90e2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 771a90e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:35:02,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cd8d47c >2018-05-29 21:35:02,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:35:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:35:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:35:02,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cd8d47c >2018-05-29 21:35:02,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:35:02,811-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [471f819c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:35:02,813-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [471f819c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ebbaa72 >2018-05-29 21:35:02,815-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [471f819c] FINISH, SetVdsStatusVDSCommand, log id: 7ebbaa72 >2018-05-29 21:35:02,848-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 471f819c, Job ID: fa0c6696-bdbe-4b43-addc-8a2ff143a4f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:35:02,853-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f88fefa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:35:02,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [471f819c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f88fefa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:35:02,866-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7728beec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:35:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7728beec] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:35:03,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:35:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4749cca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d4ee2ce >2018-05-29 21:35:03,285-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:03,286-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:03,290-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:03,291-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:03,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:35:03,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4749cca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:35:03,299-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4749cca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:35:03,302-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4749cca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4749cca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:35:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 62216795 >2018-05-29 21:35:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:35:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:35:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:35:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 62216795 >2018-05-29 21:35:03,403-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:35:03,416-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [33767cf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:35:03,418-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33767cf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42e99742 >2018-05-29 21:35:03,420-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [33767cf5] FINISH, SetVdsStatusVDSCommand, log id: 42e99742 >2018-05-29 21:35:03,425-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33767cf5, Job ID: 0c265509-aaa6-4306-a528-805b9e953fca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:35:03,430-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5335aa18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:35:03,432-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [33767cf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5335aa18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:35:03,442-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [46b6edc4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:35:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [46b6edc4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [11dfbe93] Autorecovering 2 hosts >2018-05-29 21:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [11dfbe93] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [53abf973] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2080f57a >2018-05-29 21:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [53abf973] FINISH, SetVdsStatusVDSCommand, log id: 2080f57a >2018-05-29 21:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [53abf973] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [53abf973] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70bda03a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 413dbf6f >2018-05-29 21:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [70bda03a] FINISH, SetVdsStatusVDSCommand, log id: 413dbf6f >2018-05-29 21:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [70bda03a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4a155340 >2018-05-29 21:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [70bda03a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4a155340 >2018-05-29 21:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [70bda03a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:40:00,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [50278757] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7eae43d0 >2018-05-29 21:40:00,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [50278757] FINISH, GetHardwareInfoVDSCommand, log id: 7eae43d0 >2018-05-29 21:40:00,551-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4779f2ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:40:00,559-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [19405a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:40:00,565-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e5c8c5b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:40:00,578-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e5c8c5b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5a2edb5a >2018-05-29 21:40:00,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:40:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [3e5c8c5b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5a2edb5a >2018-05-29 21:40:00,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:00,969-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:00,973-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:00,974-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:00,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:00,978-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [3e5c8c5b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:00,982-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [3e5c8c5b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:40:00,985-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3e5c8c5b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e5c8c5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:40:00,992-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 11ebe761 >2018-05-29 21:40:00,992-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:40:01,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7489c281] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64768251 >2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:40:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:40:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:40:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 11ebe761 >2018-05-29 21:40:01,090-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:40:01,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7489c281] FINISH, GetHardwareInfoVDSCommand, log id: 64768251 >2018-05-29 21:40:01,105-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [163e0325] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:40:01,106-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [6107ce39] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:40:01,107-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6107ce39] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d233f7d >2018-05-29 21:40:01,109-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6107ce39] FINISH, SetVdsStatusVDSCommand, log id: 2d233f7d >2018-05-29 21:40:01,114-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7f497a91] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:40:01,119-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [54d21994] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:40:01,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [54d21994] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 14bcf709 >2018-05-29 21:40:01,143-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6107ce39, Job ID: b8b4ee37-3eb9-4dc4-a225-abe8a532424f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:40:01,147-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 19405a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:40:01,149-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6107ce39] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 19405a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:40:01,160-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [13817581] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:40:01,161-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [13817581] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:40:01,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:40:01,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [54d21994] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 14bcf709 >2018-05-29 21:40:01,507-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:01,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:01,512-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:01,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:01,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:40:01,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [54d21994] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:40:01,520-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [54d21994] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:40:01,522-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54d21994] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d21994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:40:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 513d7d63 >2018-05-29 21:40:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:40:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:40:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:40:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 513d7d63 >2018-05-29 21:40:01,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:40:01,640-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [6e2cd4ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:40:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e2cd4ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32fa64da >2018-05-29 21:40:01,643-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6e2cd4ac] FINISH, SetVdsStatusVDSCommand, log id: 32fa64da >2018-05-29 21:40:01,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e2cd4ac, Job ID: 4d988bad-a574-40a3-a3ed-14c47b5d8ab4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:40:01,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f497a91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:40:01,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6e2cd4ac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f497a91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:40:01,665-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2c088b85] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:40:01,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2c088b85] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7389da8a] Autorecovering 2 hosts >2018-05-29 21:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7389da8a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [31ae02cd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e425707 >2018-05-29 21:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [31ae02cd] FINISH, SetVdsStatusVDSCommand, log id: 5e425707 >2018-05-29 21:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [31ae02cd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [31ae02cd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 626bd472 >2018-05-29 21:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] FINISH, SetVdsStatusVDSCommand, log id: 626bd472 >2018-05-29 21:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 53dcae7f >2018-05-29 21:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [5c99c35c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 53dcae7f >2018-05-29 21:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5c99c35c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:45:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [8f559e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6620f23a >2018-05-29 21:45:02,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [8f559e7] FINISH, GetHardwareInfoVDSCommand, log id: 6620f23a >2018-05-29 21:45:02,074-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [24175f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:45:02,079-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [23c4a17b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:45:02,084-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [76a31956] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:45:02,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [76a31956] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2455f89 >2018-05-29 21:45:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3fd7dbd0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 21cfb32c >2018-05-29 21:45:02,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3fd7dbd0] FINISH, GetHardwareInfoVDSCommand, log id: 21cfb32c >2018-05-29 21:45:02,406-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2422a435] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:45:02,410-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [1c2b1033] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:45:02,415-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6036e22b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:45:02,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6036e22b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 774b0816 >2018-05-29 21:45:02,466-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:45:02,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [76a31956] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2455f89 >2018-05-29 21:45:02,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,471-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,475-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [76a31956] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,482-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [76a31956] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:45:02,485-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [76a31956] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 76a31956, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:45:02,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e0af3bd >2018-05-29 21:45:02,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:45:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:45:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:45:02,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e0af3bd >2018-05-29 21:45:02,582-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:45:02,595-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6e28fe8e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:45:02,596-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e28fe8e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 34021b6a >2018-05-29 21:45:02,598-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6e28fe8e] FINISH, SetVdsStatusVDSCommand, log id: 34021b6a >2018-05-29 21:45:02,627-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e28fe8e, Job ID: 40fe12ca-f7f9-46b1-a4de-757095cf38b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:45:02,632-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 23c4a17b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:45:02,633-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6e28fe8e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 23c4a17b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:45:02,644-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6580b09f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:45:02,645-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6580b09f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:45:02,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:45:02,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6036e22b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 774b0816 >2018-05-29 21:45:02,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,796-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,801-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:45:02,805-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6036e22b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:45:02,809-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6036e22b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:45:02,811-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6036e22b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6036e22b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:45:02,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e946201 >2018-05-29 21:45:02,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:45:02,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:45:02,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:45:02,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e946201 >2018-05-29 21:45:02,912-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:45:02,925-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [534fccb7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:45:02,926-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [534fccb7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49d04571 >2018-05-29 21:45:02,928-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [534fccb7] FINISH, SetVdsStatusVDSCommand, log id: 49d04571 >2018-05-29 21:45:02,933-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 534fccb7, Job ID: 625efb00-78b0-4278-9e03-e9bbcdeb118c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:45:02,938-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c2b1033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:45:02,940-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [534fccb7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c2b1033, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:45:02,950-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [5b3d0daf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:45:02,952-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [5b3d0daf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4ab21ffa] Autorecovering 2 hosts >2018-05-29 21:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [4ab21ffa] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:50:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5e9f04a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 570a9e9e >2018-05-29 21:50:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5e9f04a7] FINISH, SetVdsStatusVDSCommand, log id: 570a9e9e >2018-05-29 21:50:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5e9f04a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:50:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5e9f04a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:50:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3333cc95 >2018-05-29 21:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] FINISH, SetVdsStatusVDSCommand, log id: 3333cc95 >2018-05-29 21:50:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 414c7fcc >2018-05-29 21:50:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4391eeb9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 414c7fcc >2018-05-29 21:50:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:50:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4391eeb9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:50:00,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [77b048da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18d8854c >2018-05-29 21:50:00,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [77b048da] FINISH, GetHardwareInfoVDSCommand, log id: 18d8854c >2018-05-29 21:50:00,241-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1c98227c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:50:00,250-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [62ca2f6d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:50:00,256-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5041cd15] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:50:00,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5041cd15] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26827452 >2018-05-29 21:50:00,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [404db175] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 55ed9423 >2018-05-29 21:50:00,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [404db175] FINISH, GetHardwareInfoVDSCommand, log id: 55ed9423 >2018-05-29 21:50:00,640-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1a28b118] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:50:00,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:50:00,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5041cd15] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26827452 >2018-05-29 21:50:00,649-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15cd92b0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:50:00,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:00,653-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:00,655-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [73e3908e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:50:00,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:00,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:00,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:00,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5041cd15] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:00,666-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5041cd15] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:50:00,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [73e3908e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 62b7d0b8 >2018-05-29 21:50:00,670-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5041cd15] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5041cd15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:50:00,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23513564 >2018-05-29 21:50:00,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:50:00,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:50:00,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:50:00,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23513564 >2018-05-29 21:50:00,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:50:00,791-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [19313a95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:50:00,792-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [19313a95] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a3b0d2d >2018-05-29 21:50:00,794-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [19313a95] FINISH, SetVdsStatusVDSCommand, log id: 7a3b0d2d >2018-05-29 21:50:00,825-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 19313a95, Job ID: abd9997b-0647-4480-9fcd-546d8e8ac667, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:50:00,830-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62ca2f6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:50:00,832-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [19313a95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62ca2f6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:50:00,843-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [50f97f50] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:50:00,844-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [50f97f50] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:50:01,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:50:01,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [73e3908e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 62b7d0b8 >2018-05-29 21:50:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:01,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:01,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:50:01,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [73e3908e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:50:01,065-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [73e3908e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:50:01,067-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [73e3908e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 73e3908e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:50:01,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51522633 >2018-05-29 21:50:01,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:50:01,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:50:01,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:50:01,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:50:01,172-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:50:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51522633 >2018-05-29 21:50:01,172-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:50:01,184-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7e89c08f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:50:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e89c08f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32f73e0 >2018-05-29 21:50:01,188-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e89c08f] FINISH, SetVdsStatusVDSCommand, log id: 32f73e0 >2018-05-29 21:50:01,194-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7e89c08f, Job ID: ad89ad7d-d3a8-41ca-b3ca-eabc1a9d7acb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:50:01,199-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15cd92b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:50:01,201-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7e89c08f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15cd92b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:50:01,211-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [40bdf9ea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:50:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [40bdf9ea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:54:43,320-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [48133d8e] Attempting to update VMs/Templates Ovf. >2018-05-29 21:54:43,322-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 21:54:43,322-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 21:54:43,323-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:54:43,327-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 21:54:43,331-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 21:54:43,331-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 21:54:43,331-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 21:54:43,331-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 21:54:43,331-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 21:54:43,332-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [50011b3e] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 21:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35e023c] Autorecovering 2 hosts >2018-05-29 21:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35e023c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:55:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:55:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:55:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2b07cc64] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7392f111 >2018-05-29 21:55:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2b07cc64] FINISH, SetVdsStatusVDSCommand, log id: 7392f111 >2018-05-29 21:55:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [2b07cc64] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 21:55:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [2b07cc64] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:55:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 21:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f6c74c6 >2018-05-29 21:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] FINISH, SetVdsStatusVDSCommand, log id: 1f6c74c6 >2018-05-29 21:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6293a85e >2018-05-29 21:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [498ec7ae] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6293a85e >2018-05-29 21:55:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 21:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [498ec7ae] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 21:55:01,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [52421d76] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: d2ea9e2 >2018-05-29 21:55:01,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [52421d76] FINISH, GetHardwareInfoVDSCommand, log id: d2ea9e2 >2018-05-29 21:55:01,579-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6bce0808] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:55:01,587-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [2d966bbf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:55:01,592-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c8cf6dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:55:01,598-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c8cf6dd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a7e4415 >2018-05-29 21:55:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6db8c0ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6ed279bc >2018-05-29 21:55:01,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6db8c0ac] FINISH, GetHardwareInfoVDSCommand, log id: 6ed279bc >2018-05-29 21:55:01,896-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [719093df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:55:01,901-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [5799d144] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:55:01,907-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [655a668] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 21:55:01,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [655a668] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f7badda >2018-05-29 21:55:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:55:01,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6c8cf6dd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a7e4415 >2018-05-29 21:55:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:01,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:01,988-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:01,989-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6c8cf6dd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:01,991-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6c8cf6dd] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:55:01,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6c8cf6dd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c8cf6dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:55:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45d66a51 >2018-05-29 21:55:02,001-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:55:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:55:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:55:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45d66a51 >2018-05-29 21:55:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:55:02,113-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5bb48a6e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:55:02,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5bb48a6e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45347fc7 >2018-05-29 21:55:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5bb48a6e] FINISH, SetVdsStatusVDSCommand, log id: 45347fc7 >2018-05-29 21:55:02,147-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bb48a6e, Job ID: 445628d2-10be-4e3a-b406-85e0eaa07092, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:55:02,152-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d966bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:55:02,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5bb48a6e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d966bbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:55:02,165-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2975f9db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 21:55:02,166-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2975f9db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 21:55:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 21:55:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [655a668] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f7badda >2018-05-29 21:55:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:03,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:03,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 21:55:03,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [655a668] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 21:55:03,297-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [655a668] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 21:55:03,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [655a668] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 655a668, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 21:55:03,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2de24871 >2018-05-29 21:55:03,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:55:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 21:55:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 21:55:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2de24871 >2018-05-29 21:55:03,400-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 21:55:03,413-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2c803754] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:55:03,414-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2c803754] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49ff4554 >2018-05-29 21:55:03,416-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2c803754] FINISH, SetVdsStatusVDSCommand, log id: 49ff4554 >2018-05-29 21:55:03,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c803754, Job ID: 0d8b9f97-8b19-4b23-a48c-71d4cf3b2397, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 21:55:03,426-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5799d144, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 21:55:03,428-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c803754] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5799d144, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 21:55:03,438-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [43509fe1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 21:55:03,439-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [43509fe1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5fe19c0d] Autorecovering 2 hosts >2018-05-29 22:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5fe19c0d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:00:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1d551005] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ef2f1c6 >2018-05-29 22:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1d551005] FINISH, SetVdsStatusVDSCommand, log id: 2ef2f1c6 >2018-05-29 22:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1d551005] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:00:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1d551005] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:00:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 546b3ab4 >2018-05-29 22:00:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] FINISH, SetVdsStatusVDSCommand, log id: 546b3ab4 >2018-05-29 22:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f2dbe3e >2018-05-29 22:00:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [58c3d40d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f2dbe3e >2018-05-29 22:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [58c3d40d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:00:01,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4a8ba379] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 266a8c61 >2018-05-29 22:00:01,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4a8ba379] FINISH, GetHardwareInfoVDSCommand, log id: 266a8c61 >2018-05-29 22:00:01,142-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [2b183378] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:00:01,146-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4b41cba8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:00:01,151-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5c6bd7f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:00:01,163-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5c6bd7f6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15ea7034 >2018-05-29 22:00:01,534-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:00:01,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5c6bd7f6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15ea7034 >2018-05-29 22:00:01,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:01,539-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:01,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:01,544-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:01,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:01,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5c6bd7f6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:01,551-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5c6bd7f6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:00:01,553-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5c6bd7f6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c6bd7f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:00:01,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4419ec3 >2018-05-29 22:00:01,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:00:01,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:00:01,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:00:01,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4419ec3 >2018-05-29 22:00:01,659-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:00:01,673-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [548d4b01] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:00:01,674-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [548d4b01] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74f12d25 >2018-05-29 22:00:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [548d4b01] FINISH, SetVdsStatusVDSCommand, log id: 74f12d25 >2018-05-29 22:00:01,684-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 548d4b01, Job ID: f6d7619f-1a20-4dbc-9973-11e2be15c6de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:00:01,689-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b41cba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:00:01,691-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [548d4b01] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b41cba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:00:01,701-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3ed29a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:00:01,702-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3ed29a3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:00:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [142c523f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e937e99 >2018-05-29 22:00:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [142c523f] FINISH, GetHardwareInfoVDSCommand, log id: 3e937e99 >2018-05-29 22:00:02,918-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [20296a1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:00:02,922-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [3194d285] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:00:02,927-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [44472301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:00:02,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [44472301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7140928f >2018-05-29 22:00:03,311-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:00:03,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [44472301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7140928f >2018-05-29 22:00:03,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:03,315-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:03,320-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:00:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [44472301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:00:03,327-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [44472301] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:00:03,330-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [44472301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 44472301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:00:03,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 20bf9e50 >2018-05-29 22:00:03,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:00:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:00:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:00:03,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:00:03,430-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:00:03,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 20bf9e50 >2018-05-29 22:00:03,430-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:00:03,442-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [89f2323] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:00:03,444-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [89f2323] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41cf5651 >2018-05-29 22:00:03,446-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [89f2323] FINISH, SetVdsStatusVDSCommand, log id: 41cf5651 >2018-05-29 22:00:03,451-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 89f2323, Job ID: 301409aa-4c33-4e92-97fa-495f6adb33f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:00:03,456-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3194d285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:00:03,458-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [89f2323] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3194d285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:00:03,468-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [67a5edd6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:00:03,469-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [67a5edd6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [156d4a8a] Autorecovering 2 hosts >2018-05-29 22:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [156d4a8a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:05:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7f90b4f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71ef797d >2018-05-29 22:05:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7f90b4f2] FINISH, SetVdsStatusVDSCommand, log id: 71ef797d >2018-05-29 22:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7f90b4f2] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:05:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f90b4f2] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:05:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [528904f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6404265e >2018-05-29 22:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [528904f9] FINISH, SetVdsStatusVDSCommand, log id: 6404265e >2018-05-29 22:05:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [528904f9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 346ab594 >2018-05-29 22:05:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [528904f9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 346ab594 >2018-05-29 22:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:05:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [528904f9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:05:01,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16a5e932] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 33f7b4bd >2018-05-29 22:05:01,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [16a5e932] FINISH, GetHardwareInfoVDSCommand, log id: 33f7b4bd >2018-05-29 22:05:01,115-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4f41b8ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:05:01,124-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [77882535] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:05:01,129-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [82aef05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:05:01,140-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [82aef05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 75ea3437 >2018-05-29 22:05:01,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:05:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [82aef05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 75ea3437 >2018-05-29 22:05:01,532-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:01,533-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:01,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:01,538-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:01,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:01,542-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [82aef05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:01,544-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [82aef05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:05:01,547-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [82aef05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 82aef05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:05:01,553-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 171132a >2018-05-29 22:05:01,553-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:05:01,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:05:01,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:05:01,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 171132a >2018-05-29 22:05:01,652-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:05:01,666-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [22c543f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:05:01,667-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22c543f6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 296543d7 >2018-05-29 22:05:01,670-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [22c543f6] FINISH, SetVdsStatusVDSCommand, log id: 296543d7 >2018-05-29 22:05:01,702-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22c543f6, Job ID: f755f6bb-600f-4ebf-8d5f-de3192c8fe9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:05:01,707-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 77882535, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:05:01,709-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [22c543f6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 77882535, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:05:01,720-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [25ede32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:05:01,721-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [25ede32] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:05:02,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [29dc36ec] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5b678ed6 >2018-05-29 22:05:02,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [29dc36ec] FINISH, GetHardwareInfoVDSCommand, log id: 5b678ed6 >2018-05-29 22:05:02,480-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [da2764f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:05:02,485-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2604d6ff] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:05:02,490-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [20d9ce58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:05:02,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [20d9ce58] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b4df2ae >2018-05-29 22:05:02,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:05:02,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [20d9ce58] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b4df2ae >2018-05-29 22:05:02,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:02,865-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:02,869-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:02,870-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:02,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:05:02,874-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [20d9ce58] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:05:02,877-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [20d9ce58] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:05:02,879-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [20d9ce58] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 20d9ce58, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:05:02,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45de65ac >2018-05-29 22:05:02,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:05:02,979-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:05:02,979-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:05:02,979-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45de65ac >2018-05-29 22:05:02,979-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:05:02,991-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5dfddcbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:05:02,992-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5dfddcbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50aeb859 >2018-05-29 22:05:02,994-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5dfddcbc] FINISH, SetVdsStatusVDSCommand, log id: 50aeb859 >2018-05-29 22:05:02,999-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5dfddcbc, Job ID: f8389a33-14fd-481d-9922-747944862cf8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:05:03,004-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2604d6ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:05:03,006-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5dfddcbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2604d6ff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:05:03,016-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [47246590] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:05:03,017-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [47246590] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [ba6a01c] Autorecovering 2 hosts >2018-05-29 22:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [ba6a01c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:10:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3860f186] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42f00e8b >2018-05-29 22:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3860f186] FINISH, SetVdsStatusVDSCommand, log id: 42f00e8b >2018-05-29 22:10:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3860f186] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:10:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3860f186] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:10:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1251800f >2018-05-29 22:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] FINISH, SetVdsStatusVDSCommand, log id: 1251800f >2018-05-29 22:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73ebfbc7 >2018-05-29 22:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [2cf44ad0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73ebfbc7 >2018-05-29 22:10:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:10:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [2cf44ad0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:10:00,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79e1c6de] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: eca7720 >2018-05-29 22:10:00,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [79e1c6de] FINISH, GetHardwareInfoVDSCommand, log id: eca7720 >2018-05-29 22:10:00,694-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [64bebb96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:10:00,702-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6d6fabfa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:10:00,707-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51519e1a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:10:00,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51519e1a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38561538 >2018-05-29 22:10:01,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:10:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [51519e1a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38561538 >2018-05-29 22:10:01,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:01,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:01,104-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:01,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:01,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:01,109-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [51519e1a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:01,112-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [51519e1a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:10:01,115-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [51519e1a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 51519e1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:10:01,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 854c251 >2018-05-29 22:10:01,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:10:01,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:10:01,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:10:01,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 854c251 >2018-05-29 22:10:01,218-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:10:01,231-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [29b35015] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:10:01,233-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29b35015] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 333bb7b >2018-05-29 22:10:01,235-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29b35015] FINISH, SetVdsStatusVDSCommand, log id: 333bb7b >2018-05-29 22:10:01,265-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29b35015, Job ID: 56185d3c-6075-4d31-8e6e-7f030c763e01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:10:01,270-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6d6fabfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:10:01,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29b35015] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6d6fabfa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:10:01,282-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [63d58cb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:10:01,284-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [63d58cb2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:10:02,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [24e38675] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11049ee9 >2018-05-29 22:10:02,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [24e38675] FINISH, GetHardwareInfoVDSCommand, log id: 11049ee9 >2018-05-29 22:10:02,358-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [c0d14b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:10:02,366-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [49732b8d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:10:02,371-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [71d2fcbe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:10:02,377-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [71d2fcbe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 158dc11d >2018-05-29 22:10:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:10:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [71d2fcbe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 158dc11d >2018-05-29 22:10:02,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:02,753-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:02,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:02,757-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:02,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:10:02,762-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [71d2fcbe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:10:02,764-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [71d2fcbe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:10:02,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [71d2fcbe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 71d2fcbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:10:02,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 34cacbd8 >2018-05-29 22:10:02,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:10:02,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:10:02,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:10:02,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 34cacbd8 >2018-05-29 22:10:02,865-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:10:02,878-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [29505c50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:10:02,879-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29505c50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21a24fbf >2018-05-29 22:10:02,881-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [29505c50] FINISH, SetVdsStatusVDSCommand, log id: 21a24fbf >2018-05-29 22:10:02,887-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29505c50, Job ID: eb927b01-8dee-4d1e-81f2-17bb8939bfc9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:10:02,891-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 49732b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:10:02,893-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [29505c50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 49732b8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:10:02,904-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5505e11c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:10:02,905-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5505e11c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1e91abfb] Autorecovering 2 hosts >2018-05-29 22:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1e91abfb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [116c8874] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 223e3204 >2018-05-29 22:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [116c8874] FINISH, SetVdsStatusVDSCommand, log id: 223e3204 >2018-05-29 22:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [116c8874] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:15:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [116c8874] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:15:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1cc56030] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4a95403 >2018-05-29 22:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1cc56030] FINISH, SetVdsStatusVDSCommand, log id: 4a95403 >2018-05-29 22:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [1cc56030] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1fb60029 >2018-05-29 22:15:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [1cc56030] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1fb60029 >2018-05-29 22:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:15:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [1cc56030] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:15:00,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [13859967] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 53ba3c72 >2018-05-29 22:15:00,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [13859967] FINISH, GetHardwareInfoVDSCommand, log id: 53ba3c72 >2018-05-29 22:15:00,553-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2df48586] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:15:00,558-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [17e0d6a2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:15:00,563-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [68f5eb08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:15:00,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [68f5eb08] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69727c16 >2018-05-29 22:15:01,004-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:15:01,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [68f5eb08] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69727c16 >2018-05-29 22:15:01,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:01,009-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:01,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:01,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:01,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:01,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [68f5eb08] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:01,021-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [68f5eb08] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:15:01,024-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [68f5eb08] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 68f5eb08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:15:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b45ed25 >2018-05-29 22:15:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:15:01,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:15:01,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:15:01,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b45ed25 >2018-05-29 22:15:01,128-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:15:01,142-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3f0252ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:15:01,143-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3f0252ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4698ed6c >2018-05-29 22:15:01,146-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3f0252ef] FINISH, SetVdsStatusVDSCommand, log id: 4698ed6c >2018-05-29 22:15:01,179-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f0252ef, Job ID: b8995fd8-2105-419c-97ca-f7928377cf02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:15:01,183-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17e0d6a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:15:01,186-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3f0252ef] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17e0d6a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:15:01,197-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6bee2244] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:15:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6bee2244] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:15:01,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2a7e1e97] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 672a43a2 >2018-05-29 22:15:01,911-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2a7e1e97] FINISH, GetHardwareInfoVDSCommand, log id: 672a43a2 >2018-05-29 22:15:01,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2e6bdf08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:15:01,927-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [641255dd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:15:01,932-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41043288] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:15:01,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41043288] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 53522a74 >2018-05-29 22:15:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:15:03,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [41043288] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 53522a74 >2018-05-29 22:15:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:03,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:03,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:15:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [41043288] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:15:03,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [41043288] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:15:03,339-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41043288] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41043288, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:15:03,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d4bb3b >2018-05-29 22:15:03,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:15:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:15:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:15:03,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: d4bb3b >2018-05-29 22:15:03,442-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:15:03,455-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [22350e22] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:15:03,456-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [22350e22] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eddf066 >2018-05-29 22:15:03,458-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [22350e22] FINISH, SetVdsStatusVDSCommand, log id: eddf066 >2018-05-29 22:15:03,463-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22350e22, Job ID: 1538104f-e673-40f2-8de3-790b23ce75c0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:15:03,467-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 641255dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:15:03,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [22350e22] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 641255dd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:15:03,480-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [72ea9069] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:15:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [72ea9069] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:20:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c7a5020] Autorecovering 2 hosts >2018-05-29 22:20:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1c7a5020] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:20:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:20:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:20:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c39371c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 309d3955 >2018-05-29 22:20:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6c39371c] FINISH, SetVdsStatusVDSCommand, log id: 309d3955 >2018-05-29 22:20:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6c39371c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:20:00,064-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6c39371c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:20:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7177353a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 64dc81d9 >2018-05-29 22:20:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7177353a] FINISH, SetVdsStatusVDSCommand, log id: 64dc81d9 >2018-05-29 22:20:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7177353a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1c302077 >2018-05-29 22:20:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7177353a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1c302077 >2018-05-29 22:20:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:20:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7177353a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:20:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3e914c83] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 782b2574 >2018-05-29 22:20:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [3e914c83] FINISH, GetHardwareInfoVDSCommand, log id: 782b2574 >2018-05-29 22:20:01,213-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [6c61432] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:20:01,221-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [76a38157] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:20:01,226-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [37a90c24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:20:01,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [37a90c24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2244b826 >2018-05-29 22:20:01,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:20:01,605-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [37a90c24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2244b826 >2018-05-29 22:20:01,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:01,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:01,613-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:01,614-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:01,618-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:01,618-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [37a90c24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:01,621-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [37a90c24] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:20:01,624-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [37a90c24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 37a90c24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:20:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67435aef >2018-05-29 22:20:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:20:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:20:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:20:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67435aef >2018-05-29 22:20:01,727-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:20:01,739-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4a0cae08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:20:01,740-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a0cae08] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 231a9d50 >2018-05-29 22:20:01,743-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a0cae08] FINISH, SetVdsStatusVDSCommand, log id: 231a9d50 >2018-05-29 22:20:01,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4a0cae08, Job ID: bf574c7a-98b6-4edd-a907-d10ebbcfe3fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:20:01,777-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 76a38157, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:20:01,779-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4a0cae08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 76a38157, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:20:01,790-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [2dc50a6e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:20:01,791-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2dc50a6e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:20:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [125db901] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7830d29a >2018-05-29 22:20:01,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [125db901] FINISH, GetHardwareInfoVDSCommand, log id: 7830d29a >2018-05-29 22:20:01,860-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3a156e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:20:01,864-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6a0c1de0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:20:01,870-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54f5a9a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:20:01,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54f5a9a1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55841a95 >2018-05-29 22:20:02,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:20:02,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54f5a9a1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55841a95 >2018-05-29 22:20:02,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:02,255-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:02,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:02,260-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:02,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:20:02,264-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54f5a9a1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:20:02,267-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54f5a9a1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:20:02,270-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54f5a9a1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54f5a9a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:20:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5139ad8e >2018-05-29 22:20:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:20:02,370-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:20:02,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:20:02,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:20:02,371-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:20:02,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5139ad8e >2018-05-29 22:20:02,371-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:20:02,383-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1334538a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:20:02,385-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1334538a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78d9f2e5 >2018-05-29 22:20:02,387-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1334538a] FINISH, SetVdsStatusVDSCommand, log id: 78d9f2e5 >2018-05-29 22:20:02,392-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1334538a, Job ID: 9e26fb77-ccfd-45d2-8b75-a98ca5641007, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:20:02,397-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a0c1de0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:20:02,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1334538a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a0c1de0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:20:02,409-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [555be83] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:20:02,410-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [555be83] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5cc79e73] Autorecovering 2 hosts >2018-05-29 22:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5cc79e73] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [63961d74] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36f80713 >2018-05-29 22:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [63961d74] FINISH, SetVdsStatusVDSCommand, log id: 36f80713 >2018-05-29 22:25:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [63961d74] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:25:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [63961d74] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:25:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c4de061] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42502daa >2018-05-29 22:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c4de061] FINISH, SetVdsStatusVDSCommand, log id: 42502daa >2018-05-29 22:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1c4de061] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5ad53a32 >2018-05-29 22:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [1c4de061] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5ad53a32 >2018-05-29 22:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:25:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1c4de061] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:25:02,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [679a19ae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 56b707ed >2018-05-29 22:25:02,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [679a19ae] FINISH, GetHardwareInfoVDSCommand, log id: 56b707ed >2018-05-29 22:25:02,588-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2d7075f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:25:02,597-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4a8787e3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:25:02,602-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1977ad99] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:25:02,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1977ad99] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 57b9362f >2018-05-29 22:25:03,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5543f98a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 647cb06e >2018-05-29 22:25:03,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5543f98a] FINISH, GetHardwareInfoVDSCommand, log id: 647cb06e >2018-05-29 22:25:03,016-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7b3ab04c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:25:03,021-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [34bb238e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:25:03,026-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [89ef2b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:25:03,032-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [89ef2b8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47173f07 >2018-05-29 22:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:25:03,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [89ef2b8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47173f07 >2018-05-29 22:25:03,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,416-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [89ef2b8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,419-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [89ef2b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:25:03,422-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [89ef2b8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 89ef2b8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:25:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6adc4b1 >2018-05-29 22:25:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:25:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:25:03,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:25:03,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:25:03,519-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:25:03,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6adc4b1 >2018-05-29 22:25:03,519-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:25:03,532-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7eb65b56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:25:03,533-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7eb65b56] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26988c09 >2018-05-29 22:25:03,535-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7eb65b56] FINISH, SetVdsStatusVDSCommand, log id: 26988c09 >2018-05-29 22:25:03,567-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7eb65b56, Job ID: 2faecb16-9b99-4cfa-90c6-5c541e376ba8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:25:03,571-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34bb238e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:25:03,573-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7eb65b56] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34bb238e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:25:03,584-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [383c0318] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:25:03,585-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [383c0318] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:25:03,976-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:25:03,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1977ad99] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 57b9362f >2018-05-29 22:25:03,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:25:03,990-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1977ad99] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:25:03,994-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1977ad99] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:25:03,997-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1977ad99] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1977ad99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:25:04,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1048769b >2018-05-29 22:25:04,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:25:04,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:25:04,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:25:04,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1048769b >2018-05-29 22:25:04,095-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:25:04,109-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [20b024e5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:25:04,110-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [20b024e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6980e7ee >2018-05-29 22:25:04,113-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [20b024e5] FINISH, SetVdsStatusVDSCommand, log id: 6980e7ee >2018-05-29 22:25:04,118-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20b024e5, Job ID: c22dc5e2-699d-4239-998a-7f9fc84cac06, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:25:04,122-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a8787e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:25:04,124-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [20b024e5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a8787e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:25:04,135-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [65d972be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:25:04,136-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [65d972be] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6fbaf3b6] Autorecovering 2 hosts >2018-05-29 22:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6fbaf3b6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1b302f85] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30c7c69e >2018-05-29 22:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1b302f85] FINISH, SetVdsStatusVDSCommand, log id: 30c7c69e >2018-05-29 22:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1b302f85] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:30:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1b302f85] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:30:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e05fe] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32c7799e >2018-05-29 22:30:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7e05fe] FINISH, SetVdsStatusVDSCommand, log id: 32c7799e >2018-05-29 22:30:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7e05fe] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 563baff3 >2018-05-29 22:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7e05fe] FINISH, SetHaMaintenanceModeVDSCommand, log id: 563baff3 >2018-05-29 22:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7e05fe] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:30:01,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4496f797] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 50c33d69 >2018-05-29 22:30:01,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4496f797] FINISH, GetHardwareInfoVDSCommand, log id: 50c33d69 >2018-05-29 22:30:01,283-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7fb8cc94] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:30:01,291-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [734c2566] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:30:01,296-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4e251905] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:30:01,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4e251905] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 246ebacd >2018-05-29 22:30:01,691-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:30:01,691-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4e251905] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 246ebacd >2018-05-29 22:30:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:01,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:01,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:01,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:01,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4e251905] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:01,708-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4e251905] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:30:01,711-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4e251905] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4e251905, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:30:01,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3cc8b204 >2018-05-29 22:30:01,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:30:01,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5cbbf6a6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3f707ee0 >2018-05-29 22:30:01,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5cbbf6a6] FINISH, GetHardwareInfoVDSCommand, log id: 3f707ee0 >2018-05-29 22:30:01,780-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [445b7c1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:30:01,789-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [18f00fff] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:30:01,794-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [668d6ace] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:30:01,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [668d6ace] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 663d7517 >2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:30:01,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:30:01,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:30:01,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3cc8b204 >2018-05-29 22:30:01,817-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:30:01,830-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [20d43faa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:30:01,831-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [20d43faa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ea44f86 >2018-05-29 22:30:01,834-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [20d43faa] FINISH, SetVdsStatusVDSCommand, log id: 3ea44f86 >2018-05-29 22:30:01,866-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 20d43faa, Job ID: 14ec65de-c20d-48ca-aac6-83dcdaaecfd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:30:01,872-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 734c2566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:30:01,874-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [20d43faa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 734c2566, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:30:01,885-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1f59ac02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:30:01,886-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1f59ac02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:30:02,180-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:30:02,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [668d6ace] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 663d7517 >2018-05-29 22:30:02,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:02,184-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:02,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:30:02,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [668d6ace] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:30:02,196-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [668d6ace] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:30:02,199-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [668d6ace] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 668d6ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:30:02,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4bbd737e >2018-05-29 22:30:02,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:30:02,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:30:02,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:30:02,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4bbd737e >2018-05-29 22:30:02,301-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:30:02,314-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [451c89ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:30:02,315-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [451c89ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 56263232 >2018-05-29 22:30:02,317-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [451c89ac] FINISH, SetVdsStatusVDSCommand, log id: 56263232 >2018-05-29 22:30:02,322-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 451c89ac, Job ID: 16638329-e209-4961-b0b0-3f13c15d054e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:30:02,327-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 18f00fff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:30:02,329-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [451c89ac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 18f00fff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:30:02,340-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6aa1e7ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:30:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6aa1e7ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6e8ac1d6] Autorecovering 2 hosts >2018-05-29 22:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [6e8ac1d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3003f38a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22912f58 >2018-05-29 22:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [3003f38a] FINISH, SetVdsStatusVDSCommand, log id: 22912f58 >2018-05-29 22:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [3003f38a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:35:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [3003f38a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:35:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [794732e7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3681ee13 >2018-05-29 22:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [794732e7] FINISH, SetVdsStatusVDSCommand, log id: 3681ee13 >2018-05-29 22:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [794732e7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5067b7b7 >2018-05-29 22:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [794732e7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5067b7b7 >2018-05-29 22:35:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [794732e7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:35:02,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [770ca8a2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 602d02a7 >2018-05-29 22:35:02,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [770ca8a2] FINISH, GetHardwareInfoVDSCommand, log id: 602d02a7 >2018-05-29 22:35:02,527-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [52b9e49a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:35:02,536-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [312e40d3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:35:02,542-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1df8bc39] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:35:02,549-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1df8bc39] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 799cc3f9 >2018-05-29 22:35:02,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:35:02,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1df8bc39] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 799cc3f9 >2018-05-29 22:35:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:02,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1df8bc39] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:02,951-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1df8bc39] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:35:02,954-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1df8bc39] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1df8bc39, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:35:02,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4777f285 >2018-05-29 22:35:02,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:35:03,004-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [62927ea3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1009adae >2018-05-29 22:35:03,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [62927ea3] FINISH, GetHardwareInfoVDSCommand, log id: 1009adae >2018-05-29 22:35:03,019-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [695a00a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:35:03,023-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [31cf9b54] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:35:03,028-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [63bcfca3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:35:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [63bcfca3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: cb5c05b >2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:35:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:35:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:35:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4777f285 >2018-05-29 22:35:03,055-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:35:03,069-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [aa5e1f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:35:03,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [aa5e1f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55f5a843 >2018-05-29 22:35:03,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [aa5e1f] FINISH, SetVdsStatusVDSCommand, log id: 55f5a843 >2018-05-29 22:35:03,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: aa5e1f, Job ID: 2b13c5a6-7f60-4f24-b660-bb4e3ac9e5ae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:35:03,108-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 312e40d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:35:03,110-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [aa5e1f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 312e40d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:35:03,121-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [30871601] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:35:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [30871601] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:35:04,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:35:04,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [63bcfca3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: cb5c05b >2018-05-29 22:35:04,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:04,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:04,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:04,410-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:04,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:35:04,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [63bcfca3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:35:04,418-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [63bcfca3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:35:04,421-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [63bcfca3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63bcfca3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:35:04,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 288f05df >2018-05-29 22:35:04,424-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:35:04,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:35:04,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:35:04,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 288f05df >2018-05-29 22:35:04,520-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:35:04,533-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [65a60336] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:35:04,534-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65a60336] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 273fce1 >2018-05-29 22:35:04,537-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [65a60336] FINISH, SetVdsStatusVDSCommand, log id: 273fce1 >2018-05-29 22:35:04,543-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a60336, Job ID: 244c1435-de5e-49b7-803d-9905fe9fbf47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:35:04,547-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 31cf9b54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:35:04,550-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [65a60336] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 31cf9b54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:35:04,560-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [44b73d75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:35:04,561-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [44b73d75] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:40:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [54e7535a] Autorecovering 2 hosts >2018-05-29 22:40:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [54e7535a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:40:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:40:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:40:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1a43452d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b7463d9 >2018-05-29 22:40:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1a43452d] FINISH, SetVdsStatusVDSCommand, log id: b7463d9 >2018-05-29 22:40:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1a43452d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:40:00,045-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [1a43452d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:40:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:40:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:40:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [cff14f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 629ff9 >2018-05-29 22:40:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [cff14f0] FINISH, SetVdsStatusVDSCommand, log id: 629ff9 >2018-05-29 22:40:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [cff14f0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6756b58c >2018-05-29 22:40:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [cff14f0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6756b58c >2018-05-29 22:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [cff14f0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:40:00,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2ed49ebe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6dfb4b34 >2018-05-29 22:40:00,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2ed49ebe] FINISH, GetHardwareInfoVDSCommand, log id: 6dfb4b34 >2018-05-29 22:40:00,727-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [326dd0b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:40:00,736-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [3b985519] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:40:00,741-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1ebb799a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:40:00,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1ebb799a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 85de922 >2018-05-29 22:40:01,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:40:01,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1ebb799a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 85de922 >2018-05-29 22:40:01,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:01,138-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:01,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:01,143-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:01,147-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:01,148-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1ebb799a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:01,151-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1ebb799a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:40:01,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1ebb799a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1ebb799a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:40:01,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 215c9aaf >2018-05-29 22:40:01,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:40:01,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:40:01,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:40:01,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 215c9aaf >2018-05-29 22:40:01,254-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:40:01,267-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1311c231] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:40:01,269-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1311c231] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 322fa388 >2018-05-29 22:40:01,271-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1311c231] FINISH, SetVdsStatusVDSCommand, log id: 322fa388 >2018-05-29 22:40:01,279-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1311c231, Job ID: 0fb017ce-3964-4a65-b717-60885dc6e671, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:40:01,284-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b985519, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:40:01,286-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1311c231] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b985519, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:40:01,297-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5ff43d28] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:40:01,298-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5ff43d28] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:40:02,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [56fb568b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1fe1817a >2018-05-29 22:40:02,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [56fb568b] FINISH, GetHardwareInfoVDSCommand, log id: 1fe1817a >2018-05-29 22:40:02,269-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7a17b567] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:40:02,273-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [34f88989] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:40:02,278-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63f01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:40:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63f01] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16fc8a6b >2018-05-29 22:40:02,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:40:02,657-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63f01] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16fc8a6b >2018-05-29 22:40:02,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:02,662-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:02,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:02,666-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:02,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:40:02,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63f01] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:40:02,673-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63f01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:40:02,676-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63f01] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63f01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:40:02,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1463b09e >2018-05-29 22:40:02,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:40:02,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:40:02,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:40:02,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:40:02,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:40:02,783-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1463b09e >2018-05-29 22:40:02,783-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:40:02,796-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [6f31a48a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:40:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f31a48a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1235915d >2018-05-29 22:40:02,800-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f31a48a] FINISH, SetVdsStatusVDSCommand, log id: 1235915d >2018-05-29 22:40:02,831-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6f31a48a, Job ID: 11b1ce51-f8d8-4597-8c63-00b2b75c5631, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:40:02,836-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34f88989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:40:02,838-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6f31a48a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34f88989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:40:02,849-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [686849e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:40:02,850-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [686849e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1780028e] Autorecovering 2 hosts >2018-05-29 22:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1780028e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:45:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [977435d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7cf1adef >2018-05-29 22:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [977435d] FINISH, SetVdsStatusVDSCommand, log id: 7cf1adef >2018-05-29 22:45:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [977435d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:45:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [977435d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:45:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7764e436 >2018-05-29 22:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] FINISH, SetVdsStatusVDSCommand, log id: 7764e436 >2018-05-29 22:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 158d3fc9 >2018-05-29 22:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5e0e8b65] FINISH, SetHaMaintenanceModeVDSCommand, log id: 158d3fc9 >2018-05-29 22:45:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:45:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5e0e8b65] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:45:00,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7ee6acfa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3baed269 >2018-05-29 22:45:00,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7ee6acfa] FINISH, GetHardwareInfoVDSCommand, log id: 3baed269 >2018-05-29 22:45:00,719-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [42f67f39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:45:00,727-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [29b31a80] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:45:00,732-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [520d088e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:45:00,744-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [520d088e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4b3e0d6a >2018-05-29 22:45:01,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:45:01,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [520d088e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4b3e0d6a >2018-05-29 22:45:01,121-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:01,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:01,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:01,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:01,131-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [520d088e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:01,134-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [520d088e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:45:01,137-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [520d088e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 520d088e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:45:01,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 61f36a85 >2018-05-29 22:45:01,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:45:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:45:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:45:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61f36a85 >2018-05-29 22:45:01,242-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:45:01,256-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6a8cdc55] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:45:01,257-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6a8cdc55] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c0bb236 >2018-05-29 22:45:01,259-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6a8cdc55] FINISH, SetVdsStatusVDSCommand, log id: 5c0bb236 >2018-05-29 22:45:01,290-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a8cdc55, Job ID: d56dbd96-7298-4a0c-a4e4-8bf8087abc2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:45:01,295-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29b31a80, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:45:01,297-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6a8cdc55] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29b31a80, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:45:01,308-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [33940d7d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:45:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [33940d7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:45:02,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cd8f557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7df1962a >2018-05-29 22:45:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6cd8f557] FINISH, GetHardwareInfoVDSCommand, log id: 7df1962a >2018-05-29 22:45:02,192-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3c568350] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:45:02,196-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [442cb1f1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:45:02,201-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [74e910a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:45:02,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [74e910a1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 47093e3 >2018-05-29 22:45:02,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:45:02,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [74e910a1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 47093e3 >2018-05-29 22:45:02,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:02,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:02,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:02,597-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:45:02,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [74e910a1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:45:02,604-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [74e910a1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:45:02,606-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [74e910a1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74e910a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:45:02,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 21e8ad5a >2018-05-29 22:45:02,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:45:02,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:45:02,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:45:02,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 21e8ad5a >2018-05-29 22:45:02,705-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:45:02,718-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5e8858aa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:45:02,719-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e8858aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5aa864b3 >2018-05-29 22:45:02,722-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5e8858aa] FINISH, SetVdsStatusVDSCommand, log id: 5aa864b3 >2018-05-29 22:45:02,732-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5e8858aa, Job ID: a5ef3ab4-fff7-4564-95bf-95de63558844, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:45:02,737-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 442cb1f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:45:02,739-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5e8858aa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 442cb1f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:45:02,749-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [784505dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:45:02,751-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [784505dc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [648d8011] Autorecovering 2 hosts >2018-05-29 22:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [648d8011] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:50:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b874010] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f2f2a0a >2018-05-29 22:50:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b874010] FINISH, SetVdsStatusVDSCommand, log id: 1f2f2a0a >2018-05-29 22:50:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b874010] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:50:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7b874010] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3f37e476] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50189949 >2018-05-29 22:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3f37e476] FINISH, SetVdsStatusVDSCommand, log id: 50189949 >2018-05-29 22:50:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3f37e476] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 54ad8b1d >2018-05-29 22:50:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3f37e476] FINISH, SetHaMaintenanceModeVDSCommand, log id: 54ad8b1d >2018-05-29 22:50:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:50:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3f37e476] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:50:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [34791797] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3e778e1e >2018-05-29 22:50:00,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [34791797] FINISH, GetHardwareInfoVDSCommand, log id: 3e778e1e >2018-05-29 22:50:00,366-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [108b8775] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:50:00,374-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d1a64ce] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:50:00,380-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [56dd39ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:50:00,386-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [56dd39ef] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: f8d97e4 >2018-05-29 22:50:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:50:00,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [56dd39ef] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: f8d97e4 >2018-05-29 22:50:00,771-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:00,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:00,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:00,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:00,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:00,781-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [56dd39ef] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:00,784-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [56dd39ef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:50:00,787-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [56dd39ef] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 56dd39ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:50:00,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17771ac4 >2018-05-29 22:50:00,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:50:00,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:50:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:50:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:50:00,887-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:50:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17771ac4 >2018-05-29 22:50:00,888-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:50:00,901-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [65a809fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:50:00,902-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a809fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46c3a4d9 >2018-05-29 22:50:00,904-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a809fa] FINISH, SetVdsStatusVDSCommand, log id: 46c3a4d9 >2018-05-29 22:50:00,934-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a809fa, Job ID: 5c911332-987e-4a35-9013-43e1cc8cd501, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:50:00,938-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d1a64ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:50:00,940-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a809fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d1a64ce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:50:00,951-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [578d9163] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:50:00,952-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [578d9163] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:50:02,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [15ea1d36] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4062128f >2018-05-29 22:50:02,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [15ea1d36] FINISH, GetHardwareInfoVDSCommand, log id: 4062128f >2018-05-29 22:50:02,057-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [55b6459a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:50:02,066-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5ebae99a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:50:02,071-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [9ea8561] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:50:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [9ea8561] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73cfb13 >2018-05-29 22:50:02,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:50:02,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [9ea8561] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73cfb13 >2018-05-29 22:50:02,458-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:02,459-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:02,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:02,463-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:02,467-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:50:02,468-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [9ea8561] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:50:02,470-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [9ea8561] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:50:02,473-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [9ea8561] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9ea8561, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:50:02,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 10f2d118 >2018-05-29 22:50:02,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:50:02,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:50:02,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:50:02,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:50:02,587-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:50:02,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 10f2d118 >2018-05-29 22:50:02,588-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:50:02,601-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [43e4f544] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:50:02,603-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43e4f544] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1def0f61 >2018-05-29 22:50:02,605-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [43e4f544] FINISH, SetVdsStatusVDSCommand, log id: 1def0f61 >2018-05-29 22:50:02,637-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43e4f544, Job ID: 32ad51f0-b546-43c1-9538-dc0f758c8ced, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:50:02,641-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ebae99a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:50:02,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43e4f544] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ebae99a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:50:02,655-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [599d1bca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:50:02,656-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [599d1bca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:54:43,332-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [6130f79f] Attempting to update VMs/Templates Ovf. >2018-05-29 22:54:43,335-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 22:54:43,335-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 22:54:43,335-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:54:43,383-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 22:54:43,387-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 22:54:43,387-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 22:54:43,387-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 22:54:43,387-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 22:54:43,388-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 22:54:43,389-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [2ebc997a] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 22:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [19e28d44] Autorecovering 2 hosts >2018-05-29 22:55:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [19e28d44] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:55:00,033-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7abfc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 679a8bb6 >2018-05-29 22:55:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [2df7abfc] FINISH, SetVdsStatusVDSCommand, log id: 679a8bb6 >2018-05-29 22:55:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [2df7abfc] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 22:55:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [2df7abfc] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 22:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1dc25fc >2018-05-29 22:55:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] FINISH, SetVdsStatusVDSCommand, log id: 1dc25fc >2018-05-29 22:55:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3d7f389e >2018-05-29 22:55:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [6f4ecc9a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3d7f389e >2018-05-29 22:55:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 22:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [6f4ecc9a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 22:55:00,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [66361ebc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 72a22764 >2018-05-29 22:55:00,292-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [66361ebc] FINISH, GetHardwareInfoVDSCommand, log id: 72a22764 >2018-05-29 22:55:00,305-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4f391ac9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:55:00,309-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [642bd1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:55:00,315-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [c5e5262] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:55:00,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [c5e5262] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11563dc4 >2018-05-29 22:55:00,706-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:55:00,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [c5e5262] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11563dc4 >2018-05-29 22:55:00,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:00,710-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:00,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:00,715-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:00,719-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:00,720-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [c5e5262] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:00,722-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [c5e5262] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:55:00,725-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [c5e5262] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c5e5262, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:55:00,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c6dde80 >2018-05-29 22:55:00,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:55:00,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:55:00,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:55:00,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c6dde80 >2018-05-29 22:55:00,830-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:55:00,844-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6b5fed9f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:55:00,845-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6b5fed9f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ec09f1f >2018-05-29 22:55:00,847-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6b5fed9f] FINISH, SetVdsStatusVDSCommand, log id: 3ec09f1f >2018-05-29 22:55:00,856-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b5fed9f, Job ID: 15f9eba5-9c5c-4bf7-8e0b-22b2dfed2665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:55:00,861-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 642bd1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:55:00,863-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b5fed9f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 642bd1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:55:00,874-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [110997c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 22:55:00,875-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [110997c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 22:55:01,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2f661022] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3cd6a371 >2018-05-29 22:55:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2f661022] FINISH, GetHardwareInfoVDSCommand, log id: 3cd6a371 >2018-05-29 22:55:01,549-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [48090f3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:55:01,554-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [8d64f96] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:55:01,559-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4d4ba2a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 22:55:01,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4d4ba2a3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: d65845f >2018-05-29 22:55:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 22:55:01,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4d4ba2a3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: d65845f >2018-05-29 22:55:01,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:01,954-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:01,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:01,958-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:01,962-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 22:55:01,963-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4d4ba2a3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 22:55:01,966-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4d4ba2a3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 22:55:01,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d4ba2a3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d4ba2a3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 22:55:01,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1b4334dc >2018-05-29 22:55:01,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 22:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 22:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1b4334dc >2018-05-29 22:55:02,073-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 22:55:02,086-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [69bf14b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:55:02,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69bf14b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5db48c28 >2018-05-29 22:55:02,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69bf14b7] FINISH, SetVdsStatusVDSCommand, log id: 5db48c28 >2018-05-29 22:55:02,098-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 69bf14b7, Job ID: bcf7d22f-4c3f-426a-ae55-5f8edf4c6c47, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 22:55:02,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8d64f96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 22:55:02,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69bf14b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8d64f96, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 22:55:02,115-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [31471ddf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 22:55:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [31471ddf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2fe644ce] Autorecovering 2 hosts >2018-05-29 23:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2fe644ce] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:00:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b6a02d1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 171edfdf >2018-05-29 23:00:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3b6a02d1] FINISH, SetVdsStatusVDSCommand, log id: 171edfdf >2018-05-29 23:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3b6a02d1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:00:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3b6a02d1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:00:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca07405 >2018-05-29 23:00:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] FINISH, SetVdsStatusVDSCommand, log id: 6ca07405 >2018-05-29 23:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3685d3ba >2018-05-29 23:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [7c5037b8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3685d3ba >2018-05-29 23:00:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:00:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [7c5037b8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:00:01,528-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1739f83d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5c918e95 >2018-05-29 23:00:01,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [1739f83d] FINISH, GetHardwareInfoVDSCommand, log id: 5c918e95 >2018-05-29 23:00:01,543-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5085d019] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:00:01,552-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1ced3c5d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:00:01,557-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [454263b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:00:01,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [454263b6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 17fc72e >2018-05-29 23:00:01,932-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:00:01,932-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [454263b6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 17fc72e >2018-05-29 23:00:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:01,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:01,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:01,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:01,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [454263b6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:01,949-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [454263b6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:00:01,951-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [454263b6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 454263b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:00:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28f4328 >2018-05-29 23:00:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:00:02,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:00:02,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:00:02,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28f4328 >2018-05-29 23:00:02,053-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:00:02,067-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [53299d63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:00:02,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [53299d63] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e1564b0 >2018-05-29 23:00:02,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [53299d63] FINISH, SetVdsStatusVDSCommand, log id: 5e1564b0 >2018-05-29 23:00:02,100-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 53299d63, Job ID: b5858af7-2e42-4b32-8e04-d3f693e3eff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:00:02,105-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1ced3c5d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:00:02,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [53299d63] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1ced3c5d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:00:02,118-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [da44710] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:00:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [da44710] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:00:02,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3c47fac9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 284f6cc9 >2018-05-29 23:00:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3c47fac9] FINISH, GetHardwareInfoVDSCommand, log id: 284f6cc9 >2018-05-29 23:00:02,800-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [272a546c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:00:02,805-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [706953a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:00:02,810-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [20907629] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:00:02,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [20907629] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 71af19fb >2018-05-29 23:00:03,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:00:03,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [20907629] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 71af19fb >2018-05-29 23:00:03,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:03,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:03,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:03,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:03,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:00:03,203-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [20907629] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:00:03,206-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [20907629] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:00:03,209-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [20907629] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 20907629, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:00:03,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 50dafc7 >2018-05-29 23:00:03,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:00:03,306-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:00:03,306-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:00:03,306-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 50dafc7 >2018-05-29 23:00:03,306-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:00:03,318-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [34a68da3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:00:03,320-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [34a68da3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c813ad2 >2018-05-29 23:00:03,322-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [34a68da3] FINISH, SetVdsStatusVDSCommand, log id: 1c813ad2 >2018-05-29 23:00:03,327-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 34a68da3, Job ID: 28c96c64-92cc-4f0e-b482-ca5c04b84f61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:00:03,332-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 706953a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:00:03,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [34a68da3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 706953a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:00:03,345-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [17b5e7ff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:00:03,346-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [17b5e7ff] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [429a7988] Autorecovering 2 hosts >2018-05-29 23:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [429a7988] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:05:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [158bcdee] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 561f9476 >2018-05-29 23:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [158bcdee] FINISH, SetVdsStatusVDSCommand, log id: 561f9476 >2018-05-29 23:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [158bcdee] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:05:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [158bcdee] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68b71222] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39a68c53 >2018-05-29 23:05:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [68b71222] FINISH, SetVdsStatusVDSCommand, log id: 39a68c53 >2018-05-29 23:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [68b71222] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61460479 >2018-05-29 23:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [68b71222] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61460479 >2018-05-29 23:05:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [68b71222] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:05:00,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [47f0c210] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6f8b4139 >2018-05-29 23:05:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [47f0c210] FINISH, GetHardwareInfoVDSCommand, log id: 6f8b4139 >2018-05-29 23:05:01,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [9364fb0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:05:01,009-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5a29b6f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:05:01,014-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3521198f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:05:01,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3521198f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32ac8e36 >2018-05-29 23:05:01,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:05:01,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3521198f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 32ac8e36 >2018-05-29 23:05:01,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:01,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:01,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:01,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:01,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:01,427-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3521198f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:01,429-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3521198f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:05:01,432-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3521198f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3521198f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:05:01,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a602f25 >2018-05-29 23:05:01,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:05:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:05:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:05:01,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a602f25 >2018-05-29 23:05:01,540-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:05:01,553-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [257119fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:05:01,555-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [257119fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25785063 >2018-05-29 23:05:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [257119fb] FINISH, SetVdsStatusVDSCommand, log id: 25785063 >2018-05-29 23:05:01,565-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 257119fb, Job ID: 37619be1-b9ba-45e1-83c1-1e197dea5420, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:05:01,570-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5a29b6f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:05:01,572-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [257119fb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5a29b6f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:05:01,583-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [60db34ea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:05:01,585-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [60db34ea] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:05:02,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [60b0b864] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3789074b >2018-05-29 23:05:02,822-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [60b0b864] FINISH, GetHardwareInfoVDSCommand, log id: 3789074b >2018-05-29 23:05:02,836-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [fe77d2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:05:02,840-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [4e444dbe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:05:02,845-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [30c22210] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:05:02,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [30c22210] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6044ae78 >2018-05-29 23:05:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:05:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [30c22210] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6044ae78 >2018-05-29 23:05:03,231-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:03,232-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:03,236-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:03,237-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:03,240-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:05:03,241-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [30c22210] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:05:03,244-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [30c22210] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:05:03,246-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [30c22210] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 30c22210, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:05:03,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6594eccd >2018-05-29 23:05:03,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:05:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:05:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:05:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6594eccd >2018-05-29 23:05:03,348-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:05:03,361-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [5888f42c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:05:03,362-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5888f42c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f2fc89 >2018-05-29 23:05:03,364-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5888f42c] FINISH, SetVdsStatusVDSCommand, log id: 30f2fc89 >2018-05-29 23:05:03,370-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5888f42c, Job ID: fa6675aa-43dd-4daa-869b-ad76920f2900, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:05:03,374-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e444dbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:05:03,376-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [5888f42c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e444dbe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:05:03,387-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [6b3876bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:05:03,388-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6b3876bc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4cfd9818] Autorecovering 2 hosts >2018-05-29 23:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4cfd9818] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:10:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [216a2c39] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4203ff6d >2018-05-29 23:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [216a2c39] FINISH, SetVdsStatusVDSCommand, log id: 4203ff6d >2018-05-29 23:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [216a2c39] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:10:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [216a2c39] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:10:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79d51413] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4caa3e95 >2018-05-29 23:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [79d51413] FINISH, SetVdsStatusVDSCommand, log id: 4caa3e95 >2018-05-29 23:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [79d51413] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 70eaaadb >2018-05-29 23:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [79d51413] FINISH, SetHaMaintenanceModeVDSCommand, log id: 70eaaadb >2018-05-29 23:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [79d51413] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:10:01,029-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [14132cc7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 161f261f >2018-05-29 23:10:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [14132cc7] FINISH, GetHardwareInfoVDSCommand, log id: 161f261f >2018-05-29 23:10:01,043-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1f843e3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:10:01,051-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4956cff1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:10:01,057-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4906a7e5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:10:01,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4906a7e5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2608a626 >2018-05-29 23:10:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:10:01,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4906a7e5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2608a626 >2018-05-29 23:10:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:01,445-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:01,446-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:01,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:01,451-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4906a7e5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:01,453-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4906a7e5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:10:01,456-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4906a7e5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4906a7e5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:10:01,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: a8897e7 >2018-05-29 23:10:01,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:10:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:10:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:10:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: a8897e7 >2018-05-29 23:10:01,559-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:10:01,572-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [439ec386] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:10:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [439ec386] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ced8bd >2018-05-29 23:10:01,576-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [439ec386] FINISH, SetVdsStatusVDSCommand, log id: 31ced8bd >2018-05-29 23:10:01,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 439ec386, Job ID: 44467d70-2d09-4b0f-9f49-33c63556d7d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:10:01,612-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4956cff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:10:01,614-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [439ec386] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4956cff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:10:01,625-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [23b83695] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:10:01,626-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [23b83695] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:10:02,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [115edcaa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 797de36 >2018-05-29 23:10:02,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [115edcaa] FINISH, GetHardwareInfoVDSCommand, log id: 797de36 >2018-05-29 23:10:02,238-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [d91b2eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:10:02,242-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [55bc84] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:10:02,248-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [403105e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:10:02,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [403105e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38daba93 >2018-05-29 23:10:02,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:10:02,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [403105e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38daba93 >2018-05-29 23:10:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:02,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:02,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:02,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:02,643-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:10:02,644-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [403105e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:10:02,647-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [403105e9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:10:02,649-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [403105e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 403105e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:10:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c7dec0c >2018-05-29 23:10:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:10:02,746-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:10:02,746-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:10:02,746-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c7dec0c >2018-05-29 23:10:02,746-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:10:02,760-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7aaa301f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:10:02,761-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7aaa301f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 306aa827 >2018-05-29 23:10:02,763-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7aaa301f] FINISH, SetVdsStatusVDSCommand, log id: 306aa827 >2018-05-29 23:10:02,769-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7aaa301f, Job ID: 90444b6f-1bdf-4399-8fd1-34c857c0d9fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:10:02,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55bc84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:10:02,775-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7aaa301f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55bc84, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:10:02,785-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7cc15db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:10:02,787-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7cc15db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [34c00909] Autorecovering 2 hosts >2018-05-29 23:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [34c00909] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146e8e65] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d28ee2c >2018-05-29 23:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [146e8e65] FINISH, SetVdsStatusVDSCommand, log id: 3d28ee2c >2018-05-29 23:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [146e8e65] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:15:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [146e8e65] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2675e980 >2018-05-29 23:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] FINISH, SetVdsStatusVDSCommand, log id: 2675e980 >2018-05-29 23:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6ceb8d27 >2018-05-29 23:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [30e9e6e2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6ceb8d27 >2018-05-29 23:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [30e9e6e2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:15:00,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b882be0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5a783d89 >2018-05-29 23:15:00,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5b882be0] FINISH, GetHardwareInfoVDSCommand, log id: 5a783d89 >2018-05-29 23:15:00,417-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [199d431b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:15:00,425-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3103910] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:15:00,430-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7addcfc9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:15:00,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7addcfc9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 315c8a60 >2018-05-29 23:15:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:15:00,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [7addcfc9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 315c8a60 >2018-05-29 23:15:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:00,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:00,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [7addcfc9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:00,833-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [7addcfc9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:15:00,836-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7addcfc9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7addcfc9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:15:00,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1bce395c >2018-05-29 23:15:00,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:15:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:15:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:15:00,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1bce395c >2018-05-29 23:15:00,940-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:15:00,954-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [264e36ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:15:00,955-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [264e36ab] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a81d665 >2018-05-29 23:15:00,957-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [264e36ab] FINISH, SetVdsStatusVDSCommand, log id: 7a81d665 >2018-05-29 23:15:00,989-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 264e36ab, Job ID: 5b53b42f-4b28-481d-a508-37b7ddf07964, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:15:00,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3103910, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:15:00,996-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [264e36ab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3103910, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:15:01,007-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [1854937b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:15:01,008-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1854937b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:15:02,331-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7a3dee80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 564767a8 >2018-05-29 23:15:02,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7a3dee80] FINISH, GetHardwareInfoVDSCommand, log id: 564767a8 >2018-05-29 23:15:02,345-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3cffc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:15:02,353-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6830a04] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:15:02,358-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [336a28f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:15:02,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [336a28f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6b20c4e6 >2018-05-29 23:15:02,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:15:02,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [336a28f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6b20c4e6 >2018-05-29 23:15:02,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:02,741-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:02,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:02,745-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:15:02,749-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [336a28f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:15:02,752-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [336a28f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:15:02,754-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [336a28f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 336a28f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:15:02,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19c49875 >2018-05-29 23:15:02,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:15:02,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:15:02,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:15:02,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19c49875 >2018-05-29 23:15:02,853-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:15:02,865-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [38c084d0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:15:02,866-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [38c084d0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6577f2ce >2018-05-29 23:15:02,868-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [38c084d0] FINISH, SetVdsStatusVDSCommand, log id: 6577f2ce >2018-05-29 23:15:02,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 38c084d0, Job ID: 867b872a-a246-49b1-8835-09b2a76ef64f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:15:02,902-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6830a04, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:15:02,904-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [38c084d0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6830a04, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:15:02,915-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [4dc9ea16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:15:02,916-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4dc9ea16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ce616c] Autorecovering 2 hosts >2018-05-29 23:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ce616c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:20:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c9abcb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63cf119b >2018-05-29 23:20:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [56c9abcb] FINISH, SetVdsStatusVDSCommand, log id: 63cf119b >2018-05-29 23:20:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [56c9abcb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:20:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [56c9abcb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:20:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [176238ca] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e33a99 >2018-05-29 23:20:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [176238ca] FINISH, SetVdsStatusVDSCommand, log id: 5e33a99 >2018-05-29 23:20:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [176238ca] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 403fdb1b >2018-05-29 23:20:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [176238ca] FINISH, SetHaMaintenanceModeVDSCommand, log id: 403fdb1b >2018-05-29 23:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [176238ca] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:20:00,621-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [a558a99] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1151436a >2018-05-29 23:20:00,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [a558a99] FINISH, GetHardwareInfoVDSCommand, log id: 1151436a >2018-05-29 23:20:00,635-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [797d1e0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:20:00,640-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [c56deeb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:20:00,645-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [404f6c45] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:20:00,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [404f6c45] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 69081cb7 >2018-05-29 23:20:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67ffaf4a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 29337058 >2018-05-29 23:20:01,622-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [67ffaf4a] FINISH, GetHardwareInfoVDSCommand, log id: 29337058 >2018-05-29 23:20:01,635-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [38083949] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:20:01,639-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4cfce002] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:20:01,645-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [39e53779] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:20:01,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [39e53779] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 39abbec4 >2018-05-29 23:20:02,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:20:02,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [39e53779] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 39abbec4 >2018-05-29 23:20:02,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,039-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,044-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [39e53779] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,051-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [39e53779] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:20:02,054-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [39e53779] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39e53779, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:20:02,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1565fd64 >2018-05-29 23:20:02,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:20:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:20:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [404f6c45] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 69081cb7 >2018-05-29 23:20:02,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,076-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:20:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [404f6c45] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:20:02,088-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [404f6c45] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:20:02,091-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [404f6c45] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 404f6c45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:20:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:20:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:20:02,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1565fd64 >2018-05-29 23:20:02,160-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:20:02,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 595dfbff >2018-05-29 23:20:02,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:20:02,173-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6676868f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:20:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6676868f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5764380 >2018-05-29 23:20:02,177-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6676868f] FINISH, SetVdsStatusVDSCommand, log id: 5764380 >2018-05-29 23:20:02,185-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6676868f, Job ID: 8d28deed-dea3-4db7-9d4c-75503568353c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:20:02,190-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4cfce002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:20:02,192-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6676868f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4cfce002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:20:02,202-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [634315a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:20:02,203-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [634315a2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:20:02,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:20:02,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:20:02,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 595dfbff >2018-05-29 23:20:02,260-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:20:02,272-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [3c713291] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:20:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c713291] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13ff759d >2018-05-29 23:20:02,276-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3c713291] FINISH, SetVdsStatusVDSCommand, log id: 13ff759d >2018-05-29 23:20:02,285-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c713291, Job ID: ca6f88e6-f420-4751-a4f8-84d1694b6a2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:20:02,289-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c56deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:20:02,291-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c713291] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c56deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:20:02,302-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2b10faea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:20:02,303-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2b10faea] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ebe75e6] Autorecovering 2 hosts >2018-05-29 23:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2ebe75e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:25:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6274e364] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c865971 >2018-05-29 23:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6274e364] FINISH, SetVdsStatusVDSCommand, log id: 1c865971 >2018-05-29 23:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6274e364] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:25:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6274e364] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58db5caf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62d9f030 >2018-05-29 23:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [58db5caf] FINISH, SetVdsStatusVDSCommand, log id: 62d9f030 >2018-05-29 23:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58db5caf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6eddd838 >2018-05-29 23:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [58db5caf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6eddd838 >2018-05-29 23:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [58db5caf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:25:02,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ed6f859] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 207f8e5f >2018-05-29 23:25:02,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [2ed6f859] FINISH, GetHardwareInfoVDSCommand, log id: 207f8e5f >2018-05-29 23:25:02,792-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4539c11b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:25:02,800-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [745cc989] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:25:02,805-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63ea04f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:25:02,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63ea04f6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fd0ff41 >2018-05-29 23:25:02,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [63d69cf3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6f772e42 >2018-05-29 23:25:02,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [63d69cf3] FINISH, GetHardwareInfoVDSCommand, log id: 6f772e42 >2018-05-29 23:25:03,001-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [42f60e52] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:25:03,005-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1b06ae05] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:25:03,010-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2fecabff] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:25:03,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2fecabff] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5227cd32 >2018-05-29 23:25:03,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:25:03,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [63ea04f6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fd0ff41 >2018-05-29 23:25:03,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,197-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,201-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,206-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [63ea04f6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,208-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [63ea04f6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:25:03,211-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [63ea04f6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63ea04f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:25:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 447a5282 >2018-05-29 23:25:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:25:03,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:25:03,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:25:03,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 447a5282 >2018-05-29 23:25:03,308-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:25:03,321-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [679e9bf2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:25:03,322-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [679e9bf2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bbf1ec2 >2018-05-29 23:25:03,324-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [679e9bf2] FINISH, SetVdsStatusVDSCommand, log id: 3bbf1ec2 >2018-05-29 23:25:03,354-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 679e9bf2, Job ID: 5c7aaaf8-6570-49ce-8cee-44c642264e0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:25:03,358-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 745cc989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:25:03,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [679e9bf2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 745cc989, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:25:03,371-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [3edd0556] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:25:03,372-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [3edd0556] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:25:03,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:25:03,380-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2fecabff] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5227cd32 >2018-05-29 23:25:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:25:03,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2fecabff] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:25:03,397-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2fecabff] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:25:03,400-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2fecabff] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2fecabff, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:25:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75b9211a >2018-05-29 23:25:03,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:25:03,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:25:03,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:25:03,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75b9211a >2018-05-29 23:25:03,504-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:25:03,517-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [584deb2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:25:03,519-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [584deb2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 55a3d645 >2018-05-29 23:25:03,521-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [584deb2e] FINISH, SetVdsStatusVDSCommand, log id: 55a3d645 >2018-05-29 23:25:03,526-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 584deb2e, Job ID: ee3388f2-e189-4120-ab46-0eeb3e77f9f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:25:03,531-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b06ae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:25:03,533-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [584deb2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b06ae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:25:03,544-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2d02ad05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:25:03,545-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2d02ad05] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45ec4323] Autorecovering 2 hosts >2018-05-29 23:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [45ec4323] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:30:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:30:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:30:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7c21dd18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58987790 >2018-05-29 23:30:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7c21dd18] FINISH, SetVdsStatusVDSCommand, log id: 58987790 >2018-05-29 23:30:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7c21dd18] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:30:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7c21dd18] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:30:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:30:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:30:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50b719d9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556bddf5 >2018-05-29 23:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [50b719d9] FINISH, SetVdsStatusVDSCommand, log id: 556bddf5 >2018-05-29 23:30:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [50b719d9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7ced0f6 >2018-05-29 23:30:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [50b719d9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7ced0f6 >2018-05-29 23:30:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:30:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [50b719d9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:30:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2acaea5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4998301 >2018-05-29 23:30:00,966-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2acaea5] FINISH, GetHardwareInfoVDSCommand, log id: 4998301 >2018-05-29 23:30:00,978-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [592e1a35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:30:00,987-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [24207665] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:30:00,992-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [229caa9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:30:00,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [229caa9d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 598b03ea >2018-05-29 23:30:01,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [43e45caa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 68b25925 >2018-05-29 23:30:01,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [43e45caa] FINISH, GetHardwareInfoVDSCommand, log id: 68b25925 >2018-05-29 23:30:01,192-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [5ffc4f92] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:30:01,196-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4a42e16b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:30:01,203-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2d9cbedd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:30:01,209-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2d9cbedd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e329349 >2018-05-29 23:30:01,381-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:30:01,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [229caa9d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 598b03ea >2018-05-29 23:30:01,385-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:01,386-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:01,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:01,390-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:01,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:01,394-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [229caa9d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:01,397-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [229caa9d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:30:01,399-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [229caa9d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 229caa9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:30:01,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a3b7700 >2018-05-29 23:30:01,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:30:01,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:30:01,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:30:01,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a3b7700 >2018-05-29 23:30:01,504-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:30:01,518-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [39b90157] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:30:01,519-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [39b90157] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4ca84b9c >2018-05-29 23:30:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [39b90157] FINISH, SetVdsStatusVDSCommand, log id: 4ca84b9c >2018-05-29 23:30:01,553-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 39b90157, Job ID: 4dc2f37a-dde2-4c0c-bd59-732d62db6b32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:30:01,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24207665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:30:01,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [39b90157] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24207665, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:30:01,571-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [455974ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:30:01,572-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [455974ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:30:02,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:30:02,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2d9cbedd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e329349 >2018-05-29 23:30:02,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:02,585-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:02,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:02,590-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:02,593-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:30:02,594-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2d9cbedd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:30:02,598-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2d9cbedd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:30:02,600-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2d9cbedd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d9cbedd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:30:02,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4445e592 >2018-05-29 23:30:02,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:30:02,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:30:02,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:30:02,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4445e592 >2018-05-29 23:30:02,699-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:30:02,712-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [7777937d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:30:02,714-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7777937d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e1304af >2018-05-29 23:30:02,715-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7777937d] FINISH, SetVdsStatusVDSCommand, log id: 7e1304af >2018-05-29 23:30:02,721-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7777937d, Job ID: a7eeb39b-1c88-41da-8a5d-82cb2cd833d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:30:02,725-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4a42e16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:30:02,727-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7777937d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4a42e16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:30:02,738-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [47a50b34] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:30:02,739-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [47a50b34] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [10f712b1] Autorecovering 2 hosts >2018-05-29 23:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [10f712b1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:35:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [86b99f9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3036d278 >2018-05-29 23:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [86b99f9] FINISH, SetVdsStatusVDSCommand, log id: 3036d278 >2018-05-29 23:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [86b99f9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:35:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [86b99f9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:35:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [10161227] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15630d75 >2018-05-29 23:35:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [10161227] FINISH, SetVdsStatusVDSCommand, log id: 15630d75 >2018-05-29 23:35:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [10161227] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1f69f59a >2018-05-29 23:35:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [10161227] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1f69f59a >2018-05-29 23:35:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:35:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [10161227] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:35:00,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [14763349] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4e843cb5 >2018-05-29 23:35:00,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [14763349] FINISH, GetHardwareInfoVDSCommand, log id: 4e843cb5 >2018-05-29 23:35:00,446-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [3282da15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:35:00,454-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [217f4052] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:35:00,459-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [d039538] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:35:00,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [d039538] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e12f30 >2018-05-29 23:35:00,844-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:35:00,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [d039538] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e12f30 >2018-05-29 23:35:00,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:00,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:00,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:00,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:00,857-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:00,858-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [d039538] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:00,861-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [d039538] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:35:00,863-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [d039538] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d039538, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:35:00,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d98d9e8 >2018-05-29 23:35:00,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:35:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:35:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:35:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d98d9e8 >2018-05-29 23:35:00,967-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:35:00,981-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [3d6b2487] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:35:00,982-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d6b2487] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b595b2c >2018-05-29 23:35:00,984-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3d6b2487] FINISH, SetVdsStatusVDSCommand, log id: 6b595b2c >2018-05-29 23:35:01,016-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d6b2487, Job ID: 7fbf84f7-87e5-4797-ac7d-51a1fd1bd0a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:35:01,020-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 217f4052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:35:01,023-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3d6b2487] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 217f4052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:35:01,034-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [59e33220] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:35:01,035-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [59e33220] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:35:02,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e3c806b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 36872b4c >2018-05-29 23:35:02,204-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [4e3c806b] FINISH, GetHardwareInfoVDSCommand, log id: 36872b4c >2018-05-29 23:35:02,217-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [5385c976] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:35:02,225-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [5e879d08] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:35:02,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75834642] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:35:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75834642] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 570a195f >2018-05-29 23:35:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:35:02,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [75834642] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 570a195f >2018-05-29 23:35:02,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:02,676-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:02,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:02,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:02,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:35:02,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [75834642] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:35:02,687-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [75834642] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:35:02,690-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [75834642] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75834642, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:35:02,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: af16986 >2018-05-29 23:35:02,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:35:02,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:35:02,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:35:02,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: af16986 >2018-05-29 23:35:02,793-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:35:02,806-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2a7a0ff0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:35:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a7a0ff0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eb2109a >2018-05-29 23:35:02,809-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2a7a0ff0] FINISH, SetVdsStatusVDSCommand, log id: 2eb2109a >2018-05-29 23:35:02,843-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2a7a0ff0, Job ID: 0c0674f9-6912-4753-9ccb-e26c695a8e63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:35:02,847-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e879d08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:35:02,849-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2a7a0ff0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e879d08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:35:02,861-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [6f93bc0d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:35:02,862-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [6f93bc0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea8ee91] Autorecovering 2 hosts >2018-05-29 23:40:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [ea8ee91] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:40:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f3ef80b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 65b31a49 >2018-05-29 23:40:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [3f3ef80b] FINISH, SetVdsStatusVDSCommand, log id: 65b31a49 >2018-05-29 23:40:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [3f3ef80b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [3f3ef80b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:40:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 70991d89 >2018-05-29 23:40:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] FINISH, SetVdsStatusVDSCommand, log id: 70991d89 >2018-05-29 23:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3a2e56bb >2018-05-29 23:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4bab52e0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3a2e56bb >2018-05-29 23:40:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4bab52e0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:40:00,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4c406f80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5667e0e1 >2018-05-29 23:40:00,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4c406f80] FINISH, GetHardwareInfoVDSCommand, log id: 5667e0e1 >2018-05-29 23:40:00,462-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7dc8b9ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:40:00,470-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4f2135e2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:40:00,476-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [786aaf00] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:40:00,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [786aaf00] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5be6a8be >2018-05-29 23:40:00,862-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:40:00,862-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [786aaf00] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5be6a8be >2018-05-29 23:40:00,866-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:00,867-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:00,871-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:00,872-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:00,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:00,876-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [786aaf00] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:00,879-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [786aaf00] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:40:00,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [786aaf00] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 786aaf00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:40:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7efbb47b >2018-05-29 23:40:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:40:00,980-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:40:00,980-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:40:00,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:40:00,981-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:40:00,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7efbb47b >2018-05-29 23:40:00,981-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:40:00,994-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [586793f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:40:00,995-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586793f2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19c2f532 >2018-05-29 23:40:00,997-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [586793f2] FINISH, SetVdsStatusVDSCommand, log id: 19c2f532 >2018-05-29 23:40:01,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 586793f2, Job ID: 21459b2a-56b5-40d2-9767-b0f912f45bab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:40:01,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f2135e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:40:01,033-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [586793f2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f2135e2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:40:01,045-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [562dc36c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:40:01,046-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [562dc36c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:40:01,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [10ac7aae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 385e4d34 >2018-05-29 23:40:01,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [10ac7aae] FINISH, GetHardwareInfoVDSCommand, log id: 385e4d34 >2018-05-29 23:40:01,746-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [5088f758] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:40:01,751-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7af001db] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:40:01,756-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [306c195e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:40:01,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [306c195e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b890f38 >2018-05-29 23:40:03,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:40:03,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [306c195e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b890f38 >2018-05-29 23:40:03,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:03,135-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:03,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:03,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:03,144-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:40:03,144-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [306c195e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:40:03,148-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [306c195e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:40:03,150-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [306c195e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 306c195e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:40:03,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4883c357 >2018-05-29 23:40:03,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:40:03,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:40:03,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:40:03,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4883c357 >2018-05-29 23:40:03,248-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:40:03,262-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [354bf78e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:40:03,263-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [354bf78e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5efe377f >2018-05-29 23:40:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [354bf78e] FINISH, SetVdsStatusVDSCommand, log id: 5efe377f >2018-05-29 23:40:03,296-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 354bf78e, Job ID: 59cbd78c-4924-4745-aea0-5f8ecda4bb1b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:40:03,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7af001db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:40:03,302-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [354bf78e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7af001db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:40:03,313-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6092ff92] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:40:03,314-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6092ff92] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:45:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [29aa5cc0] Autorecovering 2 hosts >2018-05-29 23:45:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [29aa5cc0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:45:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b323604] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dd08377 >2018-05-29 23:45:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b323604] FINISH, SetVdsStatusVDSCommand, log id: 7dd08377 >2018-05-29 23:45:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b323604] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:45:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5b323604] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:45:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [75f721a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bbd194d >2018-05-29 23:45:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [75f721a4] FINISH, SetVdsStatusVDSCommand, log id: 2bbd194d >2018-05-29 23:45:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [75f721a4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5a19488b >2018-05-29 23:45:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [75f721a4] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5a19488b >2018-05-29 23:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:45:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [75f721a4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:45:00,979-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d2d68f3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7070036a >2018-05-29 23:45:00,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6d2d68f3] FINISH, GetHardwareInfoVDSCommand, log id: 7070036a >2018-05-29 23:45:00,994-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [47de66ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:45:00,999-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [37d56549] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:45:01,004-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [158553bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:45:01,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [158553bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 50e4992e >2018-05-29 23:45:01,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:45:01,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [158553bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 50e4992e >2018-05-29 23:45:01,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:01,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:01,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:01,397-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:01,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:01,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [158553bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:01,404-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [158553bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:45:01,407-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [158553bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 158553bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:45:01,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 680a266 >2018-05-29 23:45:01,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:45:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:45:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:45:01,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 680a266 >2018-05-29 23:45:01,513-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:45:01,526-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4c725d0e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:45:01,527-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4c725d0e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e61c1d6 >2018-05-29 23:45:01,530-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4c725d0e] FINISH, SetVdsStatusVDSCommand, log id: 5e61c1d6 >2018-05-29 23:45:01,538-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4c725d0e, Job ID: 6f54f9db-da39-4d7c-af5a-41df99b57527, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:45:01,543-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 37d56549, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:45:01,545-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c725d0e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 37d56549, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:45:01,556-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4d1a3ff2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:45:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4d1a3ff2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:45:01,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1754d4ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 60c6afb >2018-05-29 23:45:01,598-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [1754d4ea] FINISH, GetHardwareInfoVDSCommand, log id: 60c6afb >2018-05-29 23:45:01,611-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [1939e03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:45:01,615-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [6722f966] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:45:01,620-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2975b27a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:45:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2975b27a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73ae7238 >2018-05-29 23:45:02,010-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:45:02,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2975b27a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73ae7238 >2018-05-29 23:45:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:02,014-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:02,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:02,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:02,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:45:02,024-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2975b27a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:45:02,027-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2975b27a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:45:02,030-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2975b27a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2975b27a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:45:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37973309 >2018-05-29 23:45:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:45:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:45:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:45:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37973309 >2018-05-29 23:45:02,133-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:45:02,147-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [1121a086] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:45:02,148-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1121a086] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c184491 >2018-05-29 23:45:02,151-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1121a086] FINISH, SetVdsStatusVDSCommand, log id: 3c184491 >2018-05-29 23:45:02,159-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1121a086, Job ID: fce44a74-3b66-42d9-a268-9c4a199f4226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:45:02,164-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6722f966, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:45:02,166-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1121a086] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6722f966, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:45:02,177-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [7269cb79] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:45:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [7269cb79] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5ff7375] Autorecovering 2 hosts >2018-05-29 23:50:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5ff7375] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:50:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [45cd4015] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5b51cff5 >2018-05-29 23:50:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [45cd4015] FINISH, SetVdsStatusVDSCommand, log id: 5b51cff5 >2018-05-29 23:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [45cd4015] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:50:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [45cd4015] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:50:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 510d57af >2018-05-29 23:50:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] FINISH, SetVdsStatusVDSCommand, log id: 510d57af >2018-05-29 23:50:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 214a3ab1 >2018-05-29 23:50:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [4a74cb29] FINISH, SetHaMaintenanceModeVDSCommand, log id: 214a3ab1 >2018-05-29 23:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [4a74cb29] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:50:02,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6778725e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c43e048 >2018-05-29 23:50:02,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6778725e] FINISH, GetHardwareInfoVDSCommand, log id: 1c43e048 >2018-05-29 23:50:02,394-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [23b7507b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:50:02,402-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d72a1f0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:50:02,407-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2c7f3613] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:50:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2c7f3613] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e053389 >2018-05-29 23:50:03,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [647894f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2abba7c7 >2018-05-29 23:50:03,038-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [647894f] FINISH, GetHardwareInfoVDSCommand, log id: 2abba7c7 >2018-05-29 23:50:03,051-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [22b675ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:50:03,056-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [72532b8a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:50:03,062-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6ea64f18] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:50:03,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6ea64f18] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24b51893 >2018-05-29 23:50:03,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:50:03,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6ea64f18] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24b51893 >2018-05-29 23:50:03,456-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,456-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,461-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,465-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6ea64f18] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,468-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6ea64f18] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:50:03,471-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6ea64f18] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ea64f18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:50:03,478-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e16e022 >2018-05-29 23:50:03,478-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:50:03,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:50:03,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:50:03,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-29) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e16e022 >2018-05-29 23:50:03,576-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-29) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:50:03,589-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [7fd94ba9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:50:03,591-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fd94ba9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d98d4a9 >2018-05-29 23:50:03,593-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [7fd94ba9] FINISH, SetVdsStatusVDSCommand, log id: d98d4a9 >2018-05-29 23:50:03,625-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7fd94ba9, Job ID: 37d6c997-5dcd-42e1-a509-9316bb10ea2f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:50:03,629-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 72532b8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:50:03,632-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7fd94ba9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 72532b8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:50:03,643-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [79d33512] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:50:03,644-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [79d33512] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:50:03,833-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:50:03,833-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2c7f3613] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e053389 >2018-05-29 23:50:03,837-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,838-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,842-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,843-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:50:03,847-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2c7f3613] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:50:03,850-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2c7f3613] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:50:03,853-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2c7f3613] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c7f3613, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:50:03,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1589d67a >2018-05-29 23:50:03,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:50:03,952-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:50:03,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:50:03,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:50:03,952-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:50:03,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1589d67a >2018-05-29 23:50:03,953-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:50:03,965-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [36ce228b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:50:03,966-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36ce228b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5564f2d9 >2018-05-29 23:50:03,968-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36ce228b] FINISH, SetVdsStatusVDSCommand, log id: 5564f2d9 >2018-05-29 23:50:03,974-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36ce228b, Job ID: 3d768810-b9c4-4eb8-a281-5e148f56f2e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:50:03,978-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d72a1f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:50:03,980-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36ce228b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d72a1f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:50:03,991-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1e667b69] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:50:03,992-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1e667b69] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:54:43,389-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [28c99cfe] Attempting to update VMs/Templates Ovf. >2018-05-29 23:54:43,392-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 23:54:43,392-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 23:54:43,393-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:54:43,441-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to update VM OVFs in Data Center 'Default' >2018-05-29 23:54:43,445-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully updated VM OVFs in Data Center 'Default' >2018-05-29 23:54:43,445-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to update template OVFs in Data Center 'Default' >2018-05-29 23:54:43,445-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully updated templates OVFs in Data Center 'Default' >2018-05-29 23:54:43,445-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 23:54:43,446-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-29 23:54:43,447-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [40d7b69] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-29 23:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13766868] Autorecovering 2 hosts >2018-05-29 23:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [13766868] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:55:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43af1542] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8cd26e3 >2018-05-29 23:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [43af1542] FINISH, SetVdsStatusVDSCommand, log id: 8cd26e3 >2018-05-29 23:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [43af1542] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-29 23:55:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [43af1542] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-29 23:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6d2dcffc >2018-05-29 23:55:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] FINISH, SetVdsStatusVDSCommand, log id: 6d2dcffc >2018-05-29 23:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7b2e91fb >2018-05-29 23:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [7f25ba8d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7b2e91fb >2018-05-29 23:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-29 23:55:00,103-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7f25ba8d] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 7f25ba8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-29 23:55:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [7f25ba8d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-29 23:55:01,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [119348b2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5795d815 >2018-05-29 23:55:01,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [119348b2] FINISH, GetHardwareInfoVDSCommand, log id: 5795d815 >2018-05-29 23:55:01,254-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [72cad116] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:55:01,263-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [232e5ddf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:55:01,268-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [759adb2b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:55:01,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [759adb2b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 525f7826 >2018-05-29 23:55:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2db31d46] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1fd116c1 >2018-05-29 23:55:01,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2db31d46] FINISH, GetHardwareInfoVDSCommand, log id: 1fd116c1 >2018-05-29 23:55:01,617-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [492600d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:55:01,621-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1cd643cd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:55:01,626-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1c8c5c61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-29 23:55:01,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1c8c5c61] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b1d50e >2018-05-29 23:55:01,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:55:01,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [759adb2b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 525f7826 >2018-05-29 23:55:01,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:01,669-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:01,672-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:01,673-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:01,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:01,678-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [759adb2b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:01,681-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [759adb2b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:55:01,683-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [759adb2b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 759adb2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:55:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5fa99a2f >2018-05-29 23:55:01,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:55:01,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:55:01,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:55:01,788-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5fa99a2f >2018-05-29 23:55:01,788-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:55:01,802-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [791b48fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:55:01,803-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [791b48fd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6afa846e >2018-05-29 23:55:01,805-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [791b48fd] FINISH, SetVdsStatusVDSCommand, log id: 6afa846e >2018-05-29 23:55:01,839-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 791b48fd, Job ID: e373c20d-acd3-4ee3-bd75-b7074252caeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:55:01,843-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 232e5ddf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:55:01,845-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [791b48fd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 232e5ddf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:55:01,857-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [507c3c70] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-29 23:55:01,858-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [507c3c70] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-29 23:55:01,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-29 23:55:01,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1c8c5c61] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b1d50e >2018-05-29 23:55:02,002-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:02,003-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:02,007-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:02,008-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:02,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-29 23:55:02,012-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1c8c5c61] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-29 23:55:02,015-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1c8c5c61] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-29 23:55:02,017-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1c8c5c61] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1c8c5c61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-29 23:55:02,021-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 433788de >2018-05-29 23:55:02,021-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-29 23:55:02,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:55:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-29 23:55:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-29 23:55:02,117-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-29 23:55:02,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 433788de >2018-05-29 23:55:02,117-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-29 23:55:02,130-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [24ff9e9e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:55:02,131-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24ff9e9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290e2136 >2018-05-29 23:55:02,134-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [24ff9e9e] FINISH, SetVdsStatusVDSCommand, log id: 290e2136 >2018-05-29 23:55:02,164-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 24ff9e9e, Job ID: f3aa4841-67e1-4078-93b2-5a3a59691911, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-29 23:55:02,168-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1cd643cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-29 23:55:02,170-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [24ff9e9e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1cd643cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-29 23:55:02,181-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2dc57252] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-29 23:55:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2dc57252] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:00:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [49fb93e4] Autorecovering 2 hosts >2018-05-30 00:00:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [49fb93e4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:00:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5840b2f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f228e33 >2018-05-30 00:00:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5840b2f5] FINISH, SetVdsStatusVDSCommand, log id: 3f228e33 >2018-05-30 00:00:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5840b2f5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:00:00,049-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5840b2f5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:00:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:00:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:00:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21ab197c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5118bb77 >2018-05-30 00:00:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21ab197c] FINISH, SetVdsStatusVDSCommand, log id: 5118bb77 >2018-05-30 00:00:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [21ab197c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: af6eceb >2018-05-30 00:00:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [21ab197c] FINISH, SetHaMaintenanceModeVDSCommand, log id: af6eceb >2018-05-30 00:00:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21ab197c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:00:02,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [346dc253] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b15c54a >2018-05-30 00:00:02,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [346dc253] FINISH, GetHardwareInfoVDSCommand, log id: 4b15c54a >2018-05-30 00:00:02,502-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4887976a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:00:02,506-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [45193dca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:00:02,512-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e3e8b41] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:00:02,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e3e8b41] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7143fa8 >2018-05-30 00:00:02,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7f78be] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5a18a2bb >2018-05-30 00:00:02,819-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3c7f78be] FINISH, GetHardwareInfoVDSCommand, log id: 5a18a2bb >2018-05-30 00:00:02,832-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [90811b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:00:02,836-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [70bbb93e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:00:02,841-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [42dee60b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:00:02,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [42dee60b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 61a4f432 >2018-05-30 00:00:02,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:00:02,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7e3e8b41] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7143fa8 >2018-05-30 00:00:02,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:02,903-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:02,906-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:02,907-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:02,911-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [7e3e8b41] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:02,914-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7e3e8b41] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:00:02,916-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7e3e8b41] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e3e8b41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:00:02,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2512470a >2018-05-30 00:00:02,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:00:03,018-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:00:03,018-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:00:03,018-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2512470a >2018-05-30 00:00:03,018-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:00:03,031-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [78fa3c14] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:00:03,032-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78fa3c14] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5777525e >2018-05-30 00:00:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78fa3c14] FINISH, SetVdsStatusVDSCommand, log id: 5777525e >2018-05-30 00:00:03,043-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78fa3c14, Job ID: f0269ad3-f77d-4b39-a2be-94dc40ae9abb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:00:03,047-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45193dca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:00:03,049-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [78fa3c14] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45193dca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:00:03,060-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [28b7a34a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:00:03,061-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [28b7a34a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:00:04,221-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:00:04,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [42dee60b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 61a4f432 >2018-05-30 00:00:04,225-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:04,226-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:04,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:04,230-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:04,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:00:04,234-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [42dee60b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:00:04,238-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [42dee60b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:00:04,241-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [42dee60b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42dee60b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:00:04,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7461c076 >2018-05-30 00:00:04,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:00:04,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:00:04,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:00:04,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7461c076 >2018-05-30 00:00:04,344-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:00:04,357-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3c2a3c33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:00:04,358-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3c2a3c33] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 380f685 >2018-05-30 00:00:04,360-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3c2a3c33] FINISH, SetVdsStatusVDSCommand, log id: 380f685 >2018-05-30 00:00:04,366-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c2a3c33, Job ID: ae47bf9e-954b-4ef3-a000-f1a613be23ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:00:04,371-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 70bbb93e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:00:04,373-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c2a3c33] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 70bbb93e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:00:04,384-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [51ceeeca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:00:04,385-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [51ceeeca] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [35459b94] Autorecovering 2 hosts >2018-05-30 00:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [35459b94] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:05:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1239332f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 318964c4 >2018-05-30 00:05:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1239332f] FINISH, SetVdsStatusVDSCommand, log id: 318964c4 >2018-05-30 00:05:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,073-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1239332f] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 1239332f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-30 00:05:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1239332f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:05:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1239332f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6eec63d6 >2018-05-30 00:05:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] FINISH, SetVdsStatusVDSCommand, log id: 6eec63d6 >2018-05-30 00:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 652e3909 >2018-05-30 00:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [2bc1a894] FINISH, SetHaMaintenanceModeVDSCommand, log id: 652e3909 >2018-05-30 00:05:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:05:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [2bc1a894] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:05:00,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [22ccdbc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2d6db281 >2018-05-30 00:05:00,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [22ccdbc6] FINISH, GetHardwareInfoVDSCommand, log id: 2d6db281 >2018-05-30 00:05:00,772-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1dfd46cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:05:00,780-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [35d6ac0a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:05:00,786-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [262d44b7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:05:00,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [262d44b7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1773ed6 >2018-05-30 00:05:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:05:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [262d44b7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1773ed6 >2018-05-30 00:05:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:01,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [262d44b7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:01,203-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [262d44b7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:05:01,205-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [262d44b7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 262d44b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:05:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 126d130 >2018-05-30 00:05:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 126d130 >2018-05-30 00:05:01,312-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:05:01,325-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [480bb623] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:05:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [480bb623] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c74ed33 >2018-05-30 00:05:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [480bb623] FINISH, SetVdsStatusVDSCommand, log id: 2c74ed33 >2018-05-30 00:05:01,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 480bb623, Job ID: 4d54ceac-2d6b-4541-9342-e8b7730d20a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:05:01,365-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35d6ac0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:05:01,368-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [480bb623] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35d6ac0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:05:01,378-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [34ec22bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:05:01,379-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [34ec22bc] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:05:02,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25ef22cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 134ae8d1 >2018-05-30 00:05:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [25ef22cb] FINISH, GetHardwareInfoVDSCommand, log id: 134ae8d1 >2018-05-30 00:05:02,087-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [26e8e9b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:05:02,092-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6ca9db5f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:05:02,097-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bccc1a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:05:02,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bccc1a4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b549daa >2018-05-30 00:05:02,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:05:02,478-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bccc1a4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b549daa >2018-05-30 00:05:02,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:02,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:02,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:02,487-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:05:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bccc1a4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:05:02,496-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bccc1a4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:05:02,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bccc1a4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bccc1a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:05:02,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f3d4310 >2018-05-30 00:05:02,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:05:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:05:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:05:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f3d4310 >2018-05-30 00:05:02,600-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:05:02,614-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2603ada0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:05:02,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2603ada0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5a3a1f1e >2018-05-30 00:05:02,617-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2603ada0] FINISH, SetVdsStatusVDSCommand, log id: 5a3a1f1e >2018-05-30 00:05:02,648-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2603ada0, Job ID: 8395d0c3-8da2-4dfe-ba01-a81edac744d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:05:02,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6ca9db5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:05:02,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2603ada0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6ca9db5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:05:02,665-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [525e4361] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:05:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [525e4361] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:10:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5b27ba21] Autorecovering 2 hosts >2018-05-30 00:10:00,016-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5b27ba21] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:10:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:10:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:10:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50c5fdc6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4462c083 >2018-05-30 00:10:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50c5fdc6] FINISH, SetVdsStatusVDSCommand, log id: 4462c083 >2018-05-30 00:10:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [50c5fdc6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:10:00,062-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [50c5fdc6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:10:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 93fbd56 >2018-05-30 00:10:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] FINISH, SetVdsStatusVDSCommand, log id: 93fbd56 >2018-05-30 00:10:00,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7d3e1c3b >2018-05-30 00:10:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5ebb9e62] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7d3e1c3b >2018-05-30 00:10:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5ebb9e62] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:10:00,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6b3a9c50] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42dfaec4 >2018-05-30 00:10:00,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6b3a9c50] FINISH, GetHardwareInfoVDSCommand, log id: 42dfaec4 >2018-05-30 00:10:00,578-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [76159a77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:10:00,587-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [25811c05] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:10:00,593-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [27bd1513] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:10:00,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [27bd1513] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 43a292bf >2018-05-30 00:10:01,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:10:01,029-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [27bd1513] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 43a292bf >2018-05-30 00:10:01,033-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:01,033-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:01,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:01,038-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:01,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:01,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [27bd1513] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:01,046-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [27bd1513] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:10:01,049-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27bd1513] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27bd1513, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:10:01,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6423b275 >2018-05-30 00:10:01,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:10:01,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:10:01,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:10:01,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6423b275 >2018-05-30 00:10:01,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:10:01,168-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2f77c33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:10:01,170-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f77c33] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 30f6d4cc >2018-05-30 00:10:01,172-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2f77c33] FINISH, SetVdsStatusVDSCommand, log id: 30f6d4cc >2018-05-30 00:10:01,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2f77c33, Job ID: 4ac694e7-29fd-4fe6-b8d3-4fd419b25152, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:10:01,185-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 25811c05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:10:01,187-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2f77c33] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 25811c05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:10:01,197-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [338b2f35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:10:01,198-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [338b2f35] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:10:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2c1cc807] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 9023de0 >2018-05-30 00:10:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2c1cc807] FINISH, GetHardwareInfoVDSCommand, log id: 9023de0 >2018-05-30 00:10:02,091-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [68df0059] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:10:02,096-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2872ba8c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:10:02,101-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c30bff1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:10:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c30bff1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6363c3a1 >2018-05-30 00:10:02,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:10:02,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4c30bff1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6363c3a1 >2018-05-30 00:10:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:02,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:02,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:02,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:10:02,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4c30bff1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:10:02,505-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4c30bff1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:10:02,508-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4c30bff1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c30bff1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:10:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36108025 >2018-05-30 00:10:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:10:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:10:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:10:02,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36108025 >2018-05-30 00:10:02,610-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:10:02,624-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [449efaac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:10:02,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [449efaac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f89c730 >2018-05-30 00:10:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [449efaac] FINISH, SetVdsStatusVDSCommand, log id: 7f89c730 >2018-05-30 00:10:02,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 449efaac, Job ID: 20ae1e6a-89db-4aba-84c9-e8854a5413cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:10:02,641-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2872ba8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:10:02,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [449efaac] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2872ba8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:10:02,654-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7a783d66] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:10:02,655-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7a783d66] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7b784087] Autorecovering 2 hosts >2018-05-30 00:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7b784087] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [191098e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76cbed8 >2018-05-30 00:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [191098e] FINISH, SetVdsStatusVDSCommand, log id: 76cbed8 >2018-05-30 00:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [191098e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:15:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [191098e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a266596] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 266d5d1a >2018-05-30 00:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5a266596] FINISH, SetVdsStatusVDSCommand, log id: 266d5d1a >2018-05-30 00:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5a266596] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 72419ff2 >2018-05-30 00:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5a266596] FINISH, SetHaMaintenanceModeVDSCommand, log id: 72419ff2 >2018-05-30 00:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5a266596] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:15:00,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [326eb818] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6bbce88c >2018-05-30 00:15:00,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [326eb818] FINISH, GetHardwareInfoVDSCommand, log id: 6bbce88c >2018-05-30 00:15:00,324-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2b9524ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:15:00,332-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [461f4b8f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:15:00,337-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2105ed07] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:15:00,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2105ed07] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ce6d997 >2018-05-30 00:15:00,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:15:00,730-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2105ed07] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ce6d997 >2018-05-30 00:15:00,734-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:00,735-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:00,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:00,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:00,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:00,744-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2105ed07] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:00,747-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2105ed07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:15:00,750-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2105ed07] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2105ed07, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:15:00,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1aa19b65 >2018-05-30 00:15:00,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:15:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:15:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:15:00,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1aa19b65 >2018-05-30 00:15:00,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:15:00,868-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [5180e43a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:15:00,870-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5180e43a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79c7352b >2018-05-30 00:15:00,872-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5180e43a] FINISH, SetVdsStatusVDSCommand, log id: 79c7352b >2018-05-30 00:15:00,904-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5180e43a, Job ID: 7dc966aa-099f-4384-b1bd-29aae80a5a81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:15:00,909-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 461f4b8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:15:00,911-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5180e43a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 461f4b8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:15:00,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [416f6f08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:15:00,924-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [416f6f08] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:15:01,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [55b44d22] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 243b5ec1 >2018-05-30 00:15:01,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [55b44d22] FINISH, GetHardwareInfoVDSCommand, log id: 243b5ec1 >2018-05-30 00:15:01,984-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [13ac7f2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:15:01,989-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [681bd019] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:15:01,994-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [eb9eb2d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:15:02,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [eb9eb2d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 247db37c >2018-05-30 00:15:02,375-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:15:02,375-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [eb9eb2d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 247db37c >2018-05-30 00:15:02,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:02,380-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:02,385-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:02,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:15:02,389-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [eb9eb2d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:15:02,393-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [eb9eb2d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:15:02,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [eb9eb2d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: eb9eb2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:15:02,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 48d538fc >2018-05-30 00:15:02,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:15:02,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:15:02,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:15:02,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48d538fc >2018-05-30 00:15:02,499-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:15:02,513-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [4d7f4b58] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:15:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d7f4b58] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f2edace >2018-05-30 00:15:02,517-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4d7f4b58] FINISH, SetVdsStatusVDSCommand, log id: 1f2edace >2018-05-30 00:15:02,552-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4d7f4b58, Job ID: 4e6073b9-dc2e-4556-afe3-0a5d2942bbd1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:15:02,557-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 681bd019, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:15:02,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4d7f4b58] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 681bd019, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:15:02,572-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [8442de3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:15:02,573-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [8442de3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:20:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [44db1c49] Autorecovering 2 hosts >2018-05-30 00:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [44db1c49] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:20:00,012-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:20:00,018-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:20:00,018-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,018-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,020-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [763a39a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fab73dd >2018-05-30 00:20:00,022-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [763a39a4] FINISH, SetVdsStatusVDSCommand, log id: 1fab73dd >2018-05-30 00:20:00,024-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,025-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [763a39a4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:20:00,025-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [763a39a4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:20:00,030-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:20:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:20:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7062d10] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6854351f >2018-05-30 00:20:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7062d10] FINISH, SetVdsStatusVDSCommand, log id: 6854351f >2018-05-30 00:20:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7062d10] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4137d34b >2018-05-30 00:20:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [7062d10] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4137d34b >2018-05-30 00:20:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:20:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7062d10] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:20:00,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4d7f55fc] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 686d6d0d >2018-05-30 00:20:00,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4d7f55fc] FINISH, GetHardwareInfoVDSCommand, log id: 686d6d0d >2018-05-30 00:20:00,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6a60236d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:20:00,233-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2eab8ad6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:20:00,238-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2d3c1ef7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:20:00,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2d3c1ef7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1bb0538e >2018-05-30 00:20:00,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:20:00,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2d3c1ef7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1bb0538e >2018-05-30 00:20:00,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:00,628-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:00,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:00,633-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:00,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:00,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2d3c1ef7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:00,641-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2d3c1ef7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:20:00,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2d3c1ef7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d3c1ef7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:20:00,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b224142 >2018-05-30 00:20:00,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-28) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:20:00,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:20:00,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:20:00,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b224142 >2018-05-30 00:20:00,747-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-28) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:20:00,761-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [581f905] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:20:00,762-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [581f905] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5888ebd6 >2018-05-30 00:20:00,764-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [581f905] FINISH, SetVdsStatusVDSCommand, log id: 5888ebd6 >2018-05-30 00:20:00,770-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 581f905, Job ID: ec93d87e-191a-4080-8c99-78998e1055d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:20:00,774-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2eab8ad6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:20:00,776-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [581f905] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2eab8ad6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:20:00,788-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7a95c669] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:20:00,789-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7a95c669] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:20:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5169a91f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 519a2105 >2018-05-30 00:20:01,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [5169a91f] FINISH, GetHardwareInfoVDSCommand, log id: 519a2105 >2018-05-30 00:20:01,549-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [71a9da51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:20:01,553-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1d4931d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:20:01,559-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [147f90f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:20:01,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [147f90f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: bd6ae0 >2018-05-30 00:20:01,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:20:01,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [147f90f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: bd6ae0 >2018-05-30 00:20:01,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:01,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:01,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:01,948-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:01,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:20:01,952-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [147f90f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:20:01,955-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [147f90f9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:20:01,958-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [147f90f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 147f90f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:20:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 58597787 >2018-05-30 00:20:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:20:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:20:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:20:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 58597787 >2018-05-30 00:20:02,059-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:20:02,072-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [6219d895] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:20:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6219d895] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 660da0f2 >2018-05-30 00:20:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [6219d895] FINISH, SetVdsStatusVDSCommand, log id: 660da0f2 >2018-05-30 00:20:02,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6219d895, Job ID: 9faf1465-8243-4986-8717-17b4632f8b09, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:20:02,086-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d4931d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:20:02,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6219d895] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d4931d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:20:02,099-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1e81fa54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:20:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1e81fa54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c885480] Autorecovering 2 hosts >2018-05-30 00:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2c885480] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:25:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61c81534] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d4314da >2018-05-30 00:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [61c81534] FINISH, SetVdsStatusVDSCommand, log id: 3d4314da >2018-05-30 00:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [61c81534] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:25:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [61c81534] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:25:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:25:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:25:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6248f2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b1d1075 >2018-05-30 00:25:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6248f2e] FINISH, SetVdsStatusVDSCommand, log id: 4b1d1075 >2018-05-30 00:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6248f2e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 455dd458 >2018-05-30 00:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6248f2e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 455dd458 >2018-05-30 00:25:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:25:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6248f2e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:25:01,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [486c90fb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3277b072 >2018-05-30 00:25:01,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [486c90fb] FINISH, GetHardwareInfoVDSCommand, log id: 3277b072 >2018-05-30 00:25:01,493-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [474eb013] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:25:01,501-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [61f02d6c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:25:01,506-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7a918aec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:25:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7a918aec] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1383963f >2018-05-30 00:25:01,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:25:01,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7a918aec] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1383963f >2018-05-30 00:25:01,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:01,927-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:01,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:01,932-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:01,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:01,936-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7a918aec] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:01,939-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7a918aec] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:25:01,942-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7a918aec] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7a918aec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:25:01,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d984f23 >2018-05-30 00:25:01,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:25:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:25:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:25:02,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d984f23 >2018-05-30 00:25:02,042-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:25:02,056-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [6bf6c7d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:25:02,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bf6c7d2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20bf61c2 >2018-05-30 00:25:02,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6bf6c7d2] FINISH, SetVdsStatusVDSCommand, log id: 20bf61c2 >2018-05-30 00:25:02,068-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6bf6c7d2, Job ID: 32c9e7f5-23d0-4036-bd9b-c2ca6b227070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:25:02,072-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 61f02d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:25:02,075-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6bf6c7d2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 61f02d6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:25:02,085-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [a385649] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:25:02,086-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [a385649] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:25:02,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [42f235fd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6dc32606 >2018-05-30 00:25:02,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [42f235fd] FINISH, GetHardwareInfoVDSCommand, log id: 6dc32606 >2018-05-30 00:25:02,691-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [559fc9a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:25:02,695-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [269bb070] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:25:02,700-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7036138] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:25:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7036138] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 38a84f8a >2018-05-30 00:25:03,095-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:25:03,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7036138] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 38a84f8a >2018-05-30 00:25:03,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:03,100-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:03,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:03,104-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:03,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:25:03,108-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7036138] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:25:03,111-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7036138] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:25:03,114-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7036138] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7036138, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:25:03,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3f2ceea9 >2018-05-30 00:25:03,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:25:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:25:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:25:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3f2ceea9 >2018-05-30 00:25:03,214-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:25:03,227-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [408f65a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:25:03,228-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [408f65a4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 42947d18 >2018-05-30 00:25:03,230-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [408f65a4] FINISH, SetVdsStatusVDSCommand, log id: 42947d18 >2018-05-30 00:25:03,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 408f65a4, Job ID: 295ad171-6da0-4cc7-b9d9-b7eda243b883, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:25:03,244-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 269bb070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:25:03,246-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [408f65a4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 269bb070, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:25:03,257-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [17e2c355] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:25:03,258-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [17e2c355] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [27f2e42b] Autorecovering 2 hosts >2018-05-30 00:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [27f2e42b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7ae8c1e9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 593aa642 >2018-05-30 00:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7ae8c1e9] FINISH, SetVdsStatusVDSCommand, log id: 593aa642 >2018-05-30 00:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [7ae8c1e9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:30:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ae8c1e9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [412db4c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 725b2173 >2018-05-30 00:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [412db4c7] FINISH, SetVdsStatusVDSCommand, log id: 725b2173 >2018-05-30 00:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [412db4c7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4c5465ea >2018-05-30 00:30:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [412db4c7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4c5465ea >2018-05-30 00:30:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:30:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [412db4c7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:30:00,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [466f4a46] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2a290f98 >2018-05-30 00:30:00,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [466f4a46] FINISH, GetHardwareInfoVDSCommand, log id: 2a290f98 >2018-05-30 00:30:00,943-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [28defb34] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:30:00,952-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [416821ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:30:00,957-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [47e22dd8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:30:00,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [47e22dd8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6cd03b0b >2018-05-30 00:30:01,347-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:30:01,347-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [47e22dd8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6cd03b0b >2018-05-30 00:30:01,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:01,351-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:01,355-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:01,356-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:01,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:01,360-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [47e22dd8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:01,363-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [47e22dd8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:30:01,366-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [47e22dd8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 47e22dd8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:30:01,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2efd39da >2018-05-30 00:30:01,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:30:01,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:30:01,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:30:01,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2efd39da >2018-05-30 00:30:01,472-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:30:01,487-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5f1de0b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:30:01,488-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f1de0b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31ccf66f >2018-05-30 00:30:01,491-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5f1de0b0] FINISH, SetVdsStatusVDSCommand, log id: 31ccf66f >2018-05-30 00:30:01,523-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f1de0b0, Job ID: 8f957041-a59e-4a88-9a7d-826868d6b4d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:30:01,528-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 416821ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:30:01,530-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5f1de0b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 416821ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:30:01,541-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [15a71089] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:30:01,543-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [15a71089] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:30:02,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3afbf1a2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5d051460 >2018-05-30 00:30:02,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3afbf1a2] FINISH, GetHardwareInfoVDSCommand, log id: 5d051460 >2018-05-30 00:30:02,739-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1db12efd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:30:02,743-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [29c8039f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:30:02,748-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1087d5b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:30:02,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1087d5b2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4695807d >2018-05-30 00:30:03,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:30:03,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1087d5b2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4695807d >2018-05-30 00:30:03,124-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:03,125-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:03,129-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:03,129-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:03,133-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:30:03,134-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1087d5b2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:30:03,136-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1087d5b2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:30:03,139-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1087d5b2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1087d5b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:30:03,142-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4edf40d6 >2018-05-30 00:30:03,142-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:30:03,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:30:03,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:30:03,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4edf40d6 >2018-05-30 00:30:03,243-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:30:03,256-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6567c576] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:30:03,258-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6567c576] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 793e106b >2018-05-30 00:30:03,260-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6567c576] FINISH, SetVdsStatusVDSCommand, log id: 793e106b >2018-05-30 00:30:03,266-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6567c576, Job ID: 100edb10-be06-42c8-96e5-023738eda080, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:30:03,270-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29c8039f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:30:03,273-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6567c576] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29c8039f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:30:03,284-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7a8a0056] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:30:03,285-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7a8a0056] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [350b7880] Autorecovering 2 hosts >2018-05-30 00:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [350b7880] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:35:00,011-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,019-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7d71fd62] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d0ed533 >2018-05-30 00:35:00,021-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7d71fd62] FINISH, SetVdsStatusVDSCommand, log id: d0ed533 >2018-05-30 00:35:00,022-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,023-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [7d71fd62] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:35:00,023-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7d71fd62] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:35:00,029-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:35:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:35:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3413ce54 >2018-05-30 00:35:00,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] FINISH, SetVdsStatusVDSCommand, log id: 3413ce54 >2018-05-30 00:35:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 23113d83 >2018-05-30 00:35:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [62fb5d13] FINISH, SetHaMaintenanceModeVDSCommand, log id: 23113d83 >2018-05-30 00:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:35:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [62fb5d13] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:35:00,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [145694a5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a9d2606 >2018-05-30 00:35:00,896-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [145694a5] FINISH, GetHardwareInfoVDSCommand, log id: 6a9d2606 >2018-05-30 00:35:00,908-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [28c4cd84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:35:00,912-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [17c22e11] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:35:00,938-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [c948807] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:35:00,946-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [c948807] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73f1ca1b >2018-05-30 00:35:02,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10746a0e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 78a46556 >2018-05-30 00:35:02,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [10746a0e] FINISH, GetHardwareInfoVDSCommand, log id: 78a46556 >2018-05-30 00:35:02,208-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [427b00c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:35:02,213-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2642136f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:35:02,218-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6d9fe52] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:35:02,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6d9fe52] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3dd3f382 >2018-05-30 00:35:02,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:35:02,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [c948807] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73f1ca1b >2018-05-30 00:35:02,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,326-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [c948807] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,329-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [c948807] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:35:02,331-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [c948807] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: c948807, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:35:02,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2e59b44a >2018-05-30 00:35:02,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:35:02,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:35:02,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:35:02,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2e59b44a >2018-05-30 00:35:02,430-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:35:02,444-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [156d020] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:35:02,445-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [156d020] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57938522 >2018-05-30 00:35:02,447-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [156d020] FINISH, SetVdsStatusVDSCommand, log id: 57938522 >2018-05-30 00:35:02,476-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 156d020, Job ID: 587d27d3-1b07-4a81-a207-a342815391a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:35:02,481-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17c22e11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:35:02,483-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [156d020] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17c22e11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:35:02,494-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2b1bf34a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:35:02,495-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2b1bf34a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:35:02,601-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:35:02,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [6d9fe52] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3dd3f382 >2018-05-30 00:35:02,605-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,606-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,610-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,610-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:35:02,615-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [6d9fe52] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:35:02,618-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [6d9fe52] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:35:02,621-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6d9fe52] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d9fe52, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:35:02,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28d2c38d >2018-05-30 00:35:02,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:35:02,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:35:02,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:35:02,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28d2c38d >2018-05-30 00:35:02,721-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-39) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:35:02,734-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [22f6c3fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:35:02,736-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [22f6c3fa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37e51f99 >2018-05-30 00:35:02,738-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [22f6c3fa] FINISH, SetVdsStatusVDSCommand, log id: 37e51f99 >2018-05-30 00:35:02,743-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 22f6c3fa, Job ID: e58712f2-5543-4709-9709-e09176bfe389, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:35:02,748-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2642136f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:35:02,750-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [22f6c3fa] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2642136f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:35:02,761-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5f679eb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:35:02,762-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5f679eb2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2026d6e] Autorecovering 2 hosts >2018-05-30 00:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2026d6e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4524eb5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: eebc68a >2018-05-30 00:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4524eb5b] FINISH, SetVdsStatusVDSCommand, log id: eebc68a >2018-05-30 00:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4524eb5b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4524eb5b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:40:00,131-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:40:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:40:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,137-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,139-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4056ef39 >2018-05-30 00:40:00,141-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] FINISH, SetVdsStatusVDSCommand, log id: 4056ef39 >2018-05-30 00:40:00,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5f74914 >2018-05-30 00:40:00,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [56c23ea2] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5f74914 >2018-05-30 00:40:00,146-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:40:00,147-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [56c23ea2] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:40:00,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1e9a3a59] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 23c14e0e >2018-05-30 00:40:00,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1e9a3a59] FINISH, GetHardwareInfoVDSCommand, log id: 23c14e0e >2018-05-30 00:40:00,355-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [58c86255] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:40:00,364-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2153b896] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:40:00,369-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [24a3c2e3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:40:00,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [24a3c2e3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1438d1ed >2018-05-30 00:40:00,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:40:00,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [24a3c2e3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1438d1ed >2018-05-30 00:40:00,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:00,764-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:00,768-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:00,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:00,773-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [24a3c2e3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:00,776-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [24a3c2e3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:40:00,779-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [24a3c2e3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 24a3c2e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:40:00,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6c295148 >2018-05-30 00:40:00,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:40:00,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:40:00,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:40:00,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6c295148 >2018-05-30 00:40:00,881-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:40:00,895-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [577becbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:40:00,896-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [577becbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19036db5 >2018-05-30 00:40:00,898-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [577becbc] FINISH, SetVdsStatusVDSCommand, log id: 19036db5 >2018-05-30 00:40:00,930-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 577becbc, Job ID: 2682374d-e062-460a-a9ee-494cebd6f81b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:40:00,934-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2153b896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:40:00,936-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [577becbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2153b896, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:40:00,948-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [669697fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:40:00,949-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [669697fe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:40:03,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [505c6afd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60e27f89 >2018-05-30 00:40:03,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [505c6afd] FINISH, GetHardwareInfoVDSCommand, log id: 60e27f89 >2018-05-30 00:40:03,218-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [22a15ce9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:40:03,227-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [3b12692a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:40:03,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27f6dcaf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:40:03,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27f6dcaf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65640423 >2018-05-30 00:40:03,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:40:03,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [27f6dcaf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65640423 >2018-05-30 00:40:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:03,630-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:03,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:03,635-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:03,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:40:03,639-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [27f6dcaf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:40:03,642-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [27f6dcaf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:40:03,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [27f6dcaf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27f6dcaf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:40:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f1fa9eb >2018-05-30 00:40:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:40:03,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:40:03,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:40:03,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f1fa9eb >2018-05-30 00:40:03,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:40:03,762-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2acee891] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:40:03,763-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2acee891] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26c5a9c9 >2018-05-30 00:40:03,765-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2acee891] FINISH, SetVdsStatusVDSCommand, log id: 26c5a9c9 >2018-05-30 00:40:03,796-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2acee891, Job ID: bc2a7e53-84a1-4844-9c9a-1aa84d922295, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:40:03,801-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3b12692a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:40:03,803-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2acee891] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3b12692a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:40:03,814-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [633bed06] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:40:03,815-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [633bed06] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4435b1e8] Autorecovering 2 hosts >2018-05-30 00:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [4435b1e8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:45:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [10498b9a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7b6289f >2018-05-30 00:45:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [10498b9a] FINISH, SetVdsStatusVDSCommand, log id: 7b6289f >2018-05-30 00:45:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [10498b9a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:45:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [10498b9a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:45:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [57494d52] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6655a3cb >2018-05-30 00:45:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [57494d52] FINISH, SetVdsStatusVDSCommand, log id: 6655a3cb >2018-05-30 00:45:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [57494d52] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1e8179f1 >2018-05-30 00:45:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [57494d52] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1e8179f1 >2018-05-30 00:45:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [57494d52] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:45:01,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3e51262f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a1ce103 >2018-05-30 00:45:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3e51262f] FINISH, GetHardwareInfoVDSCommand, log id: 6a1ce103 >2018-05-30 00:45:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [762b2f4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 447fc3fb >2018-05-30 00:45:01,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [762b2f4] FINISH, GetHardwareInfoVDSCommand, log id: 447fc3fb >2018-05-30 00:45:01,550-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6db648f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:45:01,556-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7bc34fbd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:45:01,559-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5f0dfff9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:45:01,564-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [65794301] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:45:01,566-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [6fac2791] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:45:01,571-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6b104ac5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:45:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [65794301] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1fcf01ab >2018-05-30 00:45:01,578-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6b104ac5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6bc1f8ad >2018-05-30 00:45:01,955-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:45:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [6b104ac5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6bc1f8ad >2018-05-30 00:45:01,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:01,960-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:01,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:01,964-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:01,968-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:01,969-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [6b104ac5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:01,972-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [6b104ac5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:45:01,974-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6b104ac5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b104ac5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:45:01,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 587b0f42 >2018-05-30 00:45:01,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:45:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:45:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:45:02,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 587b0f42 >2018-05-30 00:45:02,071-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:45:02,085-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [70c1dfd3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:45:02,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [70c1dfd3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 46a32337 >2018-05-30 00:45:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [70c1dfd3] FINISH, SetVdsStatusVDSCommand, log id: 46a32337 >2018-05-30 00:45:02,095-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 70c1dfd3, Job ID: 8e3ade66-0108-4e48-abde-b6c23be53419, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:45:02,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fac2791, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:45:02,102-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [70c1dfd3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fac2791, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:45:02,113-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [94aaa4a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:45:02,114-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [94aaa4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:45:02,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:45:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [65794301] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1fcf01ab >2018-05-30 00:45:02,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:02,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:02,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:02,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:02,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:45:02,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [65794301] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:45:02,954-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [65794301] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:45:02,956-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65794301] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65794301, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:45:02,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: ccfe6fb >2018-05-30 00:45:02,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:45:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:45:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:45:03,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: ccfe6fb >2018-05-30 00:45:03,055-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:45:03,068-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3c8bf25c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:45:03,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c8bf25c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73ea559e >2018-05-30 00:45:03,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3c8bf25c] FINISH, SetVdsStatusVDSCommand, log id: 73ea559e >2018-05-30 00:45:03,077-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c8bf25c, Job ID: 202f3eb9-3eee-49f5-9534-36650ac629b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:45:03,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f0dfff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:45:03,083-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3c8bf25c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f0dfff9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:45:03,094-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3bec7ca9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:45:03,095-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3bec7ca9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:50:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6103baad] Autorecovering 2 hosts >2018-05-30 00:50:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6103baad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:50:00,030-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:50:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:50:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [745d87c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ce46562 >2018-05-30 00:50:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [745d87c1] FINISH, SetVdsStatusVDSCommand, log id: 1ce46562 >2018-05-30 00:50:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [745d87c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:50:00,043-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [745d87c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:50:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:50:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:50:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [441b7a31] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 19b5452 >2018-05-30 00:50:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [441b7a31] FINISH, SetVdsStatusVDSCommand, log id: 19b5452 >2018-05-30 00:50:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [441b7a31] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 60a8e005 >2018-05-30 00:50:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [441b7a31] FINISH, SetHaMaintenanceModeVDSCommand, log id: 60a8e005 >2018-05-30 00:50:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:50:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [441b7a31] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:50:00,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [493b3dc4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4f1638ed >2018-05-30 00:50:00,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [493b3dc4] FINISH, GetHardwareInfoVDSCommand, log id: 4f1638ed >2018-05-30 00:50:00,794-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [6b0752a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:50:00,799-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [1f3bb1cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:50:00,805-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6ee5e1e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:50:00,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6ee5e1e9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42dfd9cc >2018-05-30 00:50:01,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:50:01,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6ee5e1e9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42dfd9cc >2018-05-30 00:50:01,189-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:01,190-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:01,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:01,194-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:01,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6ee5e1e9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:01,203-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6ee5e1e9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:50:01,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6ee5e1e9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6ee5e1e9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:50:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1f9477c2 >2018-05-30 00:50:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:50:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:50:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:50:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1f9477c2 >2018-05-30 00:50:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:50:01,323-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [3b65d568] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:50:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b65d568] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 84f289c >2018-05-30 00:50:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [3b65d568] FINISH, SetVdsStatusVDSCommand, log id: 84f289c >2018-05-30 00:50:01,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3b65d568, Job ID: 434e5584-bfb7-4fc2-9c11-accdf0464ca1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:50:01,339-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1f3bb1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:50:01,341-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3b65d568] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1f3bb1cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:50:01,352-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2dcbdebb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:50:01,353-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2dcbdebb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:50:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3e51d283] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6a581112 >2018-05-30 00:50:02,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3e51d283] FINISH, GetHardwareInfoVDSCommand, log id: 6a581112 >2018-05-30 00:50:02,748-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [63ee8e66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:50:02,753-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [8e182d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:50:02,758-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3e07120e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:50:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3e07120e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 27379b3f >2018-05-30 00:50:03,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:50:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3e07120e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 27379b3f >2018-05-30 00:50:03,152-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:03,153-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:03,156-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:03,157-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:03,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:50:03,162-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3e07120e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:50:03,164-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3e07120e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:50:03,166-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e07120e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e07120e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:50:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74654907 >2018-05-30 00:50:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:50:03,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:50:03,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:50:03,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74654907 >2018-05-30 00:50:03,268-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:50:03,280-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3a15d3b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:50:03,281-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a15d3b6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 14a6cc9d >2018-05-30 00:50:03,283-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a15d3b6] FINISH, SetVdsStatusVDSCommand, log id: 14a6cc9d >2018-05-30 00:50:03,291-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3a15d3b6, Job ID: 11353512-4cda-43f6-80c7-a3b77d825174, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:50:03,296-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 8e182d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:50:03,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3a15d3b6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 8e182d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:50:03,308-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7208bff0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:50:03,309-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7208bff0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:54:43,448-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [436482c5] Attempting to update VMs/Templates Ovf. >2018-05-30 00:54:43,451-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 00:54:43,451-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 00:54:43,451-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:54:43,456-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 00:54:43,460-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 00:54:43,460-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 00:54:43,460-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 00:54:43,460-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 00:54:43,460-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 00:54:43,461-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [147563cb] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 00:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [20accf8d] Autorecovering 2 hosts >2018-05-30 00:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [20accf8d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [9c48754] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43b6e58e >2018-05-30 00:55:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [9c48754] FINISH, SetVdsStatusVDSCommand, log id: 43b6e58e >2018-05-30 00:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [9c48754] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 00:55:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [9c48754] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:55:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 00:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66b43539] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fae1e73 >2018-05-30 00:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [66b43539] FINISH, SetVdsStatusVDSCommand, log id: 7fae1e73 >2018-05-30 00:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [66b43539] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 18ab9f2d >2018-05-30 00:55:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [66b43539] FINISH, SetHaMaintenanceModeVDSCommand, log id: 18ab9f2d >2018-05-30 00:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 00:55:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [66b43539] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 00:55:00,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [15f33d40] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 541eb4f0 >2018-05-30 00:55:00,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [15f33d40] FINISH, GetHardwareInfoVDSCommand, log id: 541eb4f0 >2018-05-30 00:55:00,989-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [77dfd55e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:55:00,997-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [20b30cf3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:55:01,002-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65242552] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:55:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65242552] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 850e8cb >2018-05-30 00:55:01,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:55:01,397-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [65242552] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 850e8cb >2018-05-30 00:55:01,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:01,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:01,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:01,407-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:01,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:01,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [65242552] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:01,414-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [65242552] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:55:01,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [65242552] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 65242552, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:55:01,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 686bb331 >2018-05-30 00:55:01,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:55:01,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:55:01,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:55:01,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 686bb331 >2018-05-30 00:55:01,520-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:55:01,534-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [68e8754b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:55:01,535-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68e8754b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1f33aa5d >2018-05-30 00:55:01,537-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [68e8754b] FINISH, SetVdsStatusVDSCommand, log id: 1f33aa5d >2018-05-30 00:55:01,569-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 68e8754b, Job ID: 743c618d-652c-446e-a79a-44609da1b052, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:55:01,574-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 20b30cf3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:55:01,576-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [68e8754b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 20b30cf3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:55:01,587-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [6efc6cc2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 00:55:01,589-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6efc6cc2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 00:55:02,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d819e94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 374582e9 >2018-05-30 00:55:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4d819e94] FINISH, GetHardwareInfoVDSCommand, log id: 374582e9 >2018-05-30 00:55:02,090-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [115eebac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:55:02,098-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1d1e6d8f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:55:02,103-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [39528918] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 00:55:02,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [39528918] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64815155 >2018-05-30 00:55:02,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 00:55:02,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [39528918] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64815155 >2018-05-30 00:55:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:02,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:02,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:02,495-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:02,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 00:55:02,500-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [39528918] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 00:55:02,502-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [39528918] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 00:55:02,505-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39528918] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 39528918, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 00:55:02,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 13020a06 >2018-05-30 00:55:02,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 00:55:02,612-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:55:02,612-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 00:55:02,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 00:55:02,613-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 00:55:02,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 13020a06 >2018-05-30 00:55:02,613-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 00:55:02,627-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4def9e8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:55:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4def9e8a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2df0bcdd >2018-05-30 00:55:02,630-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4def9e8a] FINISH, SetVdsStatusVDSCommand, log id: 2df0bcdd >2018-05-30 00:55:02,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4def9e8a, Job ID: 2d0d8e18-fa18-444a-871f-693f7f513c40, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 00:55:02,640-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d1e6d8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 00:55:02,642-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4def9e8a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d1e6d8f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 00:55:02,653-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [177d9cbf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 00:55:02,654-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [177d9cbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6a229617] Autorecovering 2 hosts >2018-05-30 01:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6a229617] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c992ff4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a22cd33 >2018-05-30 01:00:01,637-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6c992ff4] FINISH, SetVdsStatusVDSCommand, log id: 1a22cd33 >2018-05-30 01:00:01,639-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:01,640-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6c992ff4] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:00:01,640-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6c992ff4] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:00:01,647-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:00:01,653-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:00:01,653-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:01,653-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:01,655-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ff2829b >2018-05-30 01:00:01,657-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] FINISH, SetVdsStatusVDSCommand, log id: 7ff2829b >2018-05-30 01:00:01,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 8d65a94 >2018-05-30 01:00:01,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [78a3b7cc] FINISH, SetHaMaintenanceModeVDSCommand, log id: 8d65a94 >2018-05-30 01:00:01,661-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:00:01,663-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [78a3b7cc] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:00:02,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [9bae053] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5691f004 >2018-05-30 01:00:02,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [9bae053] FINISH, GetHardwareInfoVDSCommand, log id: 5691f004 >2018-05-30 01:00:02,249-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [4e466d88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:00:02,254-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [33a4da62] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:00:02,259-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5da674f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:00:02,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5da674f0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a164128 >2018-05-30 01:00:02,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:00:02,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [5da674f0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a164128 >2018-05-30 01:00:02,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:02,642-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:02,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:02,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [5da674f0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:02,654-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [5da674f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:00:02,657-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5da674f0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5da674f0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:00:02,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d9d2654 >2018-05-30 01:00:02,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:00:02,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:00:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:00:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:00:02,760-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:00:02,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d9d2654 >2018-05-30 01:00:02,760-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:00:02,774-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [30069ce0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:00:02,775-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30069ce0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b5caa82 >2018-05-30 01:00:02,777-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30069ce0] FINISH, SetVdsStatusVDSCommand, log id: b5caa82 >2018-05-30 01:00:02,808-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 30069ce0, Job ID: 1e1033ab-e385-4033-80dd-e24a75f3bc81, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:00:02,813-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33a4da62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:00:02,815-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [30069ce0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33a4da62, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:00:02,826-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [239f7d23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:00:02,827-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [239f7d23] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:00:03,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [13483ed3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4949dd81 >2018-05-30 01:00:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [13483ed3] FINISH, GetHardwareInfoVDSCommand, log id: 4949dd81 >2018-05-30 01:00:03,493-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4eed6b56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:00:03,502-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [7f6a7927] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:00:03,507-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [36bb71a5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:00:03,518-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [36bb71a5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 65a63f5d >2018-05-30 01:00:03,887-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:00:03,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [36bb71a5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 65a63f5d >2018-05-30 01:00:03,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:03,892-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:03,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:03,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:03,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:00:03,901-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [36bb71a5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:00:03,903-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [36bb71a5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:00:03,906-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36bb71a5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36bb71a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:00:03,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: d30f79a >2018-05-30 01:00:03,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:00:04,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:00:04,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:00:04,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: d30f79a >2018-05-30 01:00:04,008-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:00:04,021-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4b136d2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:00:04,022-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4b136d2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41b90a48 >2018-05-30 01:00:04,024-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4b136d2b] FINISH, SetVdsStatusVDSCommand, log id: 41b90a48 >2018-05-30 01:00:04,030-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4b136d2b, Job ID: 0a9abc53-d6ad-4cb0-bc4c-9c7df2fdf95d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:00:04,034-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7f6a7927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:00:04,036-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4b136d2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7f6a7927, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:00:04,047-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [5c100f04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:00:04,048-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5c100f04] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ee51af] Autorecovering 2 hosts >2018-05-30 01:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7ee51af] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1426e774] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39295565 >2018-05-30 01:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1426e774] FINISH, SetVdsStatusVDSCommand, log id: 39295565 >2018-05-30 01:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1426e774] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1426e774] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [574e1661] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a09240f >2018-05-30 01:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [574e1661] FINISH, SetVdsStatusVDSCommand, log id: a09240f >2018-05-30 01:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [574e1661] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 118a05f7 >2018-05-30 01:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [574e1661] FINISH, SetHaMaintenanceModeVDSCommand, log id: 118a05f7 >2018-05-30 01:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [574e1661] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:05:00,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2313b160] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2e278a51 >2018-05-30 01:05:00,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2313b160] FINISH, GetHardwareInfoVDSCommand, log id: 2e278a51 >2018-05-30 01:05:00,435-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6483e93d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:05:00,443-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [79720ad] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:05:00,448-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e373c38] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:05:00,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e373c38] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2605cd43 >2018-05-30 01:05:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:05:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1e373c38] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2605cd43 >2018-05-30 01:05:00,850-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:00,851-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:00,855-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:00,856-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:00,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:00,860-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1e373c38] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:00,863-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1e373c38] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:05:00,865-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e373c38] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1e373c38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:05:00,872-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 26ed6575 >2018-05-30 01:05:00,872-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:05:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:05:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:05:00,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 26ed6575 >2018-05-30 01:05:00,972-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:05:00,986-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3f9da000] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:05:00,987-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f9da000] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3292fd78 >2018-05-30 01:05:00,989-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f9da000] FINISH, SetVdsStatusVDSCommand, log id: 3292fd78 >2018-05-30 01:05:01,021-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f9da000, Job ID: 6983663d-e1ff-4653-877e-22a3761d9484, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:05:01,026-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79720ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:05:01,028-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f9da000] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79720ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:05:01,039-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [689e41b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:05:01,040-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [689e41b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:05:01,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ceac5c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6bb7ea4b >2018-05-30 01:05:01,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [ceac5c] FINISH, GetHardwareInfoVDSCommand, log id: 6bb7ea4b >2018-05-30 01:05:01,750-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [33fc6e30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:05:01,755-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [7cb72bd9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:05:01,760-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2cfc536f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:05:01,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2cfc536f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2dac804a >2018-05-30 01:05:03,141-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:05:03,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2cfc536f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2dac804a >2018-05-30 01:05:03,145-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:03,146-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:03,149-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:03,150-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:05:03,155-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2cfc536f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:05:03,158-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2cfc536f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:05:03,162-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cfc536f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2cfc536f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:05:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 228cf4c4 >2018-05-30 01:05:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:05:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:05:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:05:03,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 228cf4c4 >2018-05-30 01:05:03,265-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:05:03,278-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [33380958] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:05:03,280-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [33380958] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 9b77a10 >2018-05-30 01:05:03,282-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [33380958] FINISH, SetVdsStatusVDSCommand, log id: 9b77a10 >2018-05-30 01:05:03,314-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 33380958, Job ID: 943c8276-6456-447b-9107-a72b02e7be2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:05:03,319-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7cb72bd9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:05:03,321-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [33380958] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7cb72bd9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:05:03,332-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [3f6d1dfc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:05:03,333-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [3f6d1dfc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:10:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6ea69e5e] Autorecovering 2 hosts >2018-05-30 01:10:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6ea69e5e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:10:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:10:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:10:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d54a04e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c42094 >2018-05-30 01:10:00,053-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6d54a04e] FINISH, SetVdsStatusVDSCommand, log id: 4c42094 >2018-05-30 01:10:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6d54a04e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:10:00,056-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6d54a04e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:10:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54061b0a >2018-05-30 01:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] FINISH, SetVdsStatusVDSCommand, log id: 54061b0a >2018-05-30 01:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6905d62 >2018-05-30 01:10:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [396ed3b3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6905d62 >2018-05-30 01:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [396ed3b3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:10:01,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75d87e39] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 45857dc1 >2018-05-30 01:10:01,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [75d87e39] FINISH, GetHardwareInfoVDSCommand, log id: 45857dc1 >2018-05-30 01:10:01,102-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7c922630] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:10:01,111-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3d34bb5a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:10:01,115-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [420e0c82] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:10:01,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [420e0c82] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e357c58 >2018-05-30 01:10:01,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:10:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [420e0c82] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e357c58 >2018-05-30 01:10:01,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:01,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:01,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:01,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:01,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [420e0c82] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:01,510-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [420e0c82] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:10:01,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [420e0c82] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 420e0c82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:10:01,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 18ebeae9 >2018-05-30 01:10:01,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:10:01,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:10:01,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:10:01,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:10:01,615-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:10:01,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18ebeae9 >2018-05-30 01:10:01,615-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:10:01,629-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1fd32e53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:10:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1fd32e53] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2188da79 >2018-05-30 01:10:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1fd32e53] FINISH, SetVdsStatusVDSCommand, log id: 2188da79 >2018-05-30 01:10:01,662-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1fd32e53, Job ID: f9c3722b-5920-4315-9d02-6f551f3dc94a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:10:01,667-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3d34bb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:10:01,669-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1fd32e53] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3d34bb5a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:10:01,680-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6c6b191b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:10:01,681-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6c6b191b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:10:01,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1799f3e6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 339edba >2018-05-30 01:10:01,772-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1799f3e6] FINISH, GetHardwareInfoVDSCommand, log id: 339edba >2018-05-30 01:10:01,786-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [689d5056] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:10:01,790-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [24aa97af] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:10:01,795-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [507215f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:10:01,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [507215f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2954ae0b >2018-05-30 01:10:02,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:10:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [507215f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2954ae0b >2018-05-30 01:10:02,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:02,183-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:02,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:02,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:02,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:10:02,192-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [507215f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:10:02,195-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [507215f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:10:02,197-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [507215f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 507215f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:10:02,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 463c46c0 >2018-05-30 01:10:02,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:10:02,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:10:02,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:10:02,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 463c46c0 >2018-05-30 01:10:02,297-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:10:02,311-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [511f5dba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:10:02,312-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [511f5dba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47bff17d >2018-05-30 01:10:02,314-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [511f5dba] FINISH, SetVdsStatusVDSCommand, log id: 47bff17d >2018-05-30 01:10:02,320-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 511f5dba, Job ID: 3e874191-b447-447a-b0ba-962ffeca3c6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:10:02,325-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 24aa97af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:10:02,327-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [511f5dba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 24aa97af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:10:02,339-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [580767a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:10:02,340-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [580767a9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [554b8631] Autorecovering 2 hosts >2018-05-30 01:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [554b8631] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:15:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:15:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:15:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [582d0662] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29111a65 >2018-05-30 01:15:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [582d0662] FINISH, SetVdsStatusVDSCommand, log id: 29111a65 >2018-05-30 01:15:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [582d0662] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:15:00,053-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [582d0662] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:15:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:15:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:15:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72d65870] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39de022 >2018-05-30 01:15:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72d65870] FINISH, SetVdsStatusVDSCommand, log id: 39de022 >2018-05-30 01:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [72d65870] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 73dbc765 >2018-05-30 01:15:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [72d65870] FINISH, SetHaMaintenanceModeVDSCommand, log id: 73dbc765 >2018-05-30 01:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:15:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [72d65870] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:15:02,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [f99ffae] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: a0feeed >2018-05-30 01:15:02,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [f99ffae] FINISH, GetHardwareInfoVDSCommand, log id: a0feeed >2018-05-30 01:15:02,377-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [501f21e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:15:02,386-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [66f5999f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:15:02,391-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5050a073] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:15:02,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5050a073] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b17e98d >2018-05-30 01:15:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [79838e7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 382485da >2018-05-30 01:15:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [79838e7] FINISH, GetHardwareInfoVDSCommand, log id: 382485da >2018-05-30 01:15:02,919-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [48fcec1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:15:02,928-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [322952b9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:15:02,933-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [50f4076a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:15:02,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [50f4076a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b8bd964 >2018-05-30 01:15:03,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:15:03,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [50f4076a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b8bd964 >2018-05-30 01:15:03,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,329-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [50f4076a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [50f4076a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:15:03,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [50f4076a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 50f4076a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:15:03,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5f30919a >2018-05-30 01:15:03,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:15:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:15:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:15:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5f30919a >2018-05-30 01:15:03,443-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:15:03,456-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7c65bcc9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:15:03,457-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7c65bcc9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f4155c4 >2018-05-30 01:15:03,459-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7c65bcc9] FINISH, SetVdsStatusVDSCommand, log id: 5f4155c4 >2018-05-30 01:15:03,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7c65bcc9, Job ID: 3cfed5e3-d537-435e-ac56-64225f094b2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:15:03,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 322952b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:15:03,500-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7c65bcc9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 322952b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:15:03,511-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [3be61973] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:15:03,512-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [3be61973] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:15:03,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:15:03,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5050a073] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b17e98d >2018-05-30 01:15:03,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,823-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,827-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:15:03,827-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5050a073] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:15:03,831-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5050a073] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:15:03,833-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5050a073] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5050a073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:15:03,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 42355f91 >2018-05-30 01:15:03,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:15:03,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:15:03,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:15:03,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 42355f91 >2018-05-30 01:15:03,933-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:15:03,946-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [156d8645] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:15:03,947-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [156d8645] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 226ef1c7 >2018-05-30 01:15:03,949-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [156d8645] FINISH, SetVdsStatusVDSCommand, log id: 226ef1c7 >2018-05-30 01:15:03,955-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 156d8645, Job ID: 697df65c-0aed-4033-952f-66147196081a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:15:03,959-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66f5999f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:15:03,962-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [156d8645] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66f5999f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:15:03,972-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [60e0e2a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:15:03,973-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [60e0e2a7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:20:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7eb5043d] Autorecovering 2 hosts >2018-05-30 01:20:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7eb5043d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:20:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [b585c0f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f58ebb4 >2018-05-30 01:20:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [b585c0f] FINISH, SetVdsStatusVDSCommand, log id: 5f58ebb4 >2018-05-30 01:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [b585c0f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:20:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [b585c0f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:20:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a5177e >2018-05-30 01:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] FINISH, SetVdsStatusVDSCommand, log id: a5177e >2018-05-30 01:20:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36462e7 >2018-05-30 01:20:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4325cb1e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36462e7 >2018-05-30 01:20:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:20:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4325cb1e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:20:01,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4a7429c3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 363ad5f9 >2018-05-30 01:20:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4a7429c3] FINISH, GetHardwareInfoVDSCommand, log id: 363ad5f9 >2018-05-30 01:20:01,089-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [720dd4b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:20:01,098-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [854a52c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:20:01,103-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4efd1a24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:20:01,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4efd1a24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2276799c >2018-05-30 01:20:01,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:20:01,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [4efd1a24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2276799c >2018-05-30 01:20:01,506-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:01,507-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:01,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:01,511-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:01,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:01,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [4efd1a24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:01,520-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [4efd1a24] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:20:01,523-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4efd1a24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4efd1a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:20:01,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15090286 >2018-05-30 01:20:01,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:20:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:20:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:20:01,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15090286 >2018-05-30 01:20:01,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:20:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4650536c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: cfd933b >2018-05-30 01:20:01,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4650536c] FINISH, GetHardwareInfoVDSCommand, log id: cfd933b >2018-05-30 01:20:01,642-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [fa6f589] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:20:01,644-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fa6f589] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73e61b33 >2018-05-30 01:20:01,646-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fa6f589] FINISH, SetVdsStatusVDSCommand, log id: 73e61b33 >2018-05-30 01:20:01,658-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [c873ab5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:20:01,664-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4e1d5bf7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:20:01,671-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169e92e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:20:01,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169e92e6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f1f8fda >2018-05-30 01:20:01,691-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: fa6f589, Job ID: 48ad0161-3799-429c-886a-4bffaeb4e3f1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:20:01,695-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 854a52c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:20:01,698-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [fa6f589] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 854a52c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:20:01,708-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [3c109678] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:20:01,710-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [3c109678] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:20:02,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:20:02,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169e92e6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f1f8fda >2018-05-30 01:20:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:02,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:02,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:02,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:02,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:20:02,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169e92e6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:20:02,071-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169e92e6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:20:02,074-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169e92e6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 169e92e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:20:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 9398add >2018-05-30 01:20:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:20:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:20:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:20:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 9398add >2018-05-30 01:20:02,175-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:20:02,189-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [60fbcf72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:20:02,190-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [60fbcf72] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25084b83 >2018-05-30 01:20:02,192-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [60fbcf72] FINISH, SetVdsStatusVDSCommand, log id: 25084b83 >2018-05-30 01:20:02,198-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60fbcf72, Job ID: 46b604eb-8b0b-4225-87fc-94eecc51dd37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:20:02,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e1d5bf7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:20:02,205-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [60fbcf72] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e1d5bf7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:20:02,216-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [4f5f1adf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:20:02,217-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [4f5f1adf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f8dcdef] Autorecovering 2 hosts >2018-05-30 01:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f8dcdef] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b99fa1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23c2ce3f >2018-05-30 01:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5b99fa1b] FINISH, SetVdsStatusVDSCommand, log id: 23c2ce3f >2018-05-30 01:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5b99fa1b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:25:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5b99fa1b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2290e22d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d8c8321 >2018-05-30 01:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2290e22d] FINISH, SetVdsStatusVDSCommand, log id: d8c8321 >2018-05-30 01:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2290e22d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6d3e4d3c >2018-05-30 01:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [2290e22d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6d3e4d3c >2018-05-30 01:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [2290e22d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:25:02,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7fc19a1b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 20b4e7a4 >2018-05-30 01:25:02,300-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7fc19a1b] FINISH, GetHardwareInfoVDSCommand, log id: 20b4e7a4 >2018-05-30 01:25:02,314-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [4b00876d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:25:02,322-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [1896678f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:25:02,328-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2047a57b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:25:02,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2047a57b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29d7fa2e >2018-05-30 01:25:02,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:25:02,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2047a57b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29d7fa2e >2018-05-30 01:25:02,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:02,727-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:02,731-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:02,732-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:02,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:02,737-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2047a57b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:02,740-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2047a57b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:25:02,742-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2047a57b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2047a57b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:25:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 351d5e94 >2018-05-30 01:25:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:25:02,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:25:02,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:25:02,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:25:02,847-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:25:02,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 351d5e94 >2018-05-30 01:25:02,847-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:25:02,861-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [349a6866] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:25:02,862-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [349a6866] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6db8138f >2018-05-30 01:25:02,864-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [349a6866] FINISH, SetVdsStatusVDSCommand, log id: 6db8138f >2018-05-30 01:25:02,896-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 349a6866, Job ID: 48e71374-1b4b-4697-be28-bd1b8bd652b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:25:02,902-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1896678f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:25:02,904-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [349a6866] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1896678f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:25:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7cef114d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 20243f52 >2018-05-30 01:25:02,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7cef114d] FINISH, GetHardwareInfoVDSCommand, log id: 20243f52 >2018-05-30 01:25:02,916-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3e1c1aae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:25:02,917-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3e1c1aae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:25:02,918-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [76c4223c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:25:02,923-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7df81cb6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:25:02,928-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3a38307a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:25:02,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3a38307a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 486e5710 >2018-05-30 01:25:03,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:25:03,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3a38307a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 486e5710 >2018-05-30 01:25:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:03,308-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:03,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:03,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:03,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:25:03,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3a38307a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:25:03,320-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3a38307a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:25:03,323-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3a38307a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a38307a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:25:03,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1cf882be >2018-05-30 01:25:03,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:25:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:25:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:25:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1cf882be >2018-05-30 01:25:03,427-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:25:03,440-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [496be5c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:25:03,441-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [496be5c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bddbb25 >2018-05-30 01:25:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [496be5c] FINISH, SetVdsStatusVDSCommand, log id: 3bddbb25 >2018-05-30 01:25:03,475-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 496be5c, Job ID: 5d2379a2-baca-4c49-bfb6-0482b1ee88cc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:25:03,480-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7df81cb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:25:03,482-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [496be5c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7df81cb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:25:03,493-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [310ad4e5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:25:03,494-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [310ad4e5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b6c7289] Autorecovering 2 hosts >2018-05-30 01:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b6c7289] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:30:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [e2431a7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41c78cb7 >2018-05-30 01:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [e2431a7] FINISH, SetVdsStatusVDSCommand, log id: 41c78cb7 >2018-05-30 01:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [e2431a7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:30:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [e2431a7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:30:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d7e47e3 >2018-05-30 01:30:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] FINISH, SetVdsStatusVDSCommand, log id: 3d7e47e3 >2018-05-30 01:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7cac73f0 >2018-05-30 01:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [3d84dc5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7cac73f0 >2018-05-30 01:30:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [3d84dc5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:30:00,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5e3dff16] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 28951bf0 >2018-05-30 01:30:00,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5e3dff16] FINISH, GetHardwareInfoVDSCommand, log id: 28951bf0 >2018-05-30 01:30:00,536-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [37163a23] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:30:00,545-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2306491e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:30:00,550-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [27427e8d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:30:00,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [27427e8d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4bb7243a >2018-05-30 01:30:00,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:30:00,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [27427e8d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4bb7243a >2018-05-30 01:30:00,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:00,946-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:00,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:00,951-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:00,955-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:00,955-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [27427e8d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:00,958-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [27427e8d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:30:00,961-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [27427e8d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 27427e8d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:30:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b60c8d0 >2018-05-30 01:30:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:30:01,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:30:01,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:30:01,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b60c8d0 >2018-05-30 01:30:01,062-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:30:01,076-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [77c9a963] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:30:01,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [77c9a963] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27aca4ed >2018-05-30 01:30:01,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [77c9a963] FINISH, SetVdsStatusVDSCommand, log id: 27aca4ed >2018-05-30 01:30:01,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 77c9a963, Job ID: b7eec72f-9f17-461f-820c-fecb9b91496c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:30:01,093-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2306491e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:30:01,095-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77c9a963] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2306491e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:30:01,106-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7f5c58f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:30:01,107-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7f5c58f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:30:01,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [57d24394] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 691a40d4 >2018-05-30 01:30:01,152-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [57d24394] FINISH, GetHardwareInfoVDSCommand, log id: 691a40d4 >2018-05-30 01:30:01,165-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7646d366] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:30:01,170-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [29d14073] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:30:01,175-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [34cbf2af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:30:01,238-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [34cbf2af] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 516316a7 >2018-05-30 01:30:01,617-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:30:01,617-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [34cbf2af] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 516316a7 >2018-05-30 01:30:01,621-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:01,622-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:01,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:01,626-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:01,630-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:30:01,631-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [34cbf2af] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:30:01,634-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [34cbf2af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:30:01,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [34cbf2af] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 34cbf2af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:30:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 36befc2b >2018-05-30 01:30:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:30:01,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:30:01,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:30:01,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 36befc2b >2018-05-30 01:30:01,740-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:30:01,753-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [21eb1cb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:30:01,755-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [21eb1cb4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 243fa512 >2018-05-30 01:30:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [21eb1cb4] FINISH, SetVdsStatusVDSCommand, log id: 243fa512 >2018-05-30 01:30:01,763-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 21eb1cb4, Job ID: 495deb3e-0124-49c0-befb-2b3887bf6882, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:30:01,767-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29d14073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:30:01,769-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21eb1cb4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29d14073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:30:01,780-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3056928a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:30:01,781-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3056928a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [67b8bb0e] Autorecovering 2 hosts >2018-05-30 01:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [67b8bb0e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:35:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:35:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:35:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ed531bb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1081d075 >2018-05-30 01:35:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3ed531bb] FINISH, SetVdsStatusVDSCommand, log id: 1081d075 >2018-05-30 01:35:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [3ed531bb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:35:00,055-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [3ed531bb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45cbafea >2018-05-30 01:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] FINISH, SetVdsStatusVDSCommand, log id: 45cbafea >2018-05-30 01:35:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 56e10e03 >2018-05-30 01:35:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [54dd20c0] FINISH, SetHaMaintenanceModeVDSCommand, log id: 56e10e03 >2018-05-30 01:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [54dd20c0] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:35:01,743-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43c2101d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 64c5d1b0 >2018-05-30 01:35:01,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [43c2101d] FINISH, GetHardwareInfoVDSCommand, log id: 64c5d1b0 >2018-05-30 01:35:01,759-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [392df5b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:35:01,768-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [21586d13] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:35:01,772-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6a9319ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:35:01,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6a9319ed] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4634801b >2018-05-30 01:35:02,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:35:02,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6a9319ed] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4634801b >2018-05-30 01:35:02,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:02,158-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:02,163-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6a9319ed] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:02,170-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6a9319ed] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:35:02,173-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6a9319ed] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6a9319ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:35:02,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1299ae5f >2018-05-30 01:35:02,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:35:02,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:35:02,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:35:02,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1299ae5f >2018-05-30 01:35:02,274-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:35:02,287-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [404ad4f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:35:02,288-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [404ad4f5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12351a1d >2018-05-30 01:35:02,290-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [404ad4f5] FINISH, SetVdsStatusVDSCommand, log id: 12351a1d >2018-05-30 01:35:02,321-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 404ad4f5, Job ID: 75301892-3963-4bd2-a87d-146f80e99420, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:35:02,326-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21586d13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:35:02,329-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [404ad4f5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21586d13, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:35:02,340-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [77d9a378] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:35:02,341-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [77d9a378] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:35:02,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7aa2a974] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1a9b4b1 >2018-05-30 01:35:02,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [7aa2a974] FINISH, GetHardwareInfoVDSCommand, log id: 1a9b4b1 >2018-05-30 01:35:02,548-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [301aee4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:35:02,552-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [401dd8f9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:35:02,557-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5c87be8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:35:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5c87be8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7cbbae8 >2018-05-30 01:35:03,931-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:35:03,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5c87be8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7cbbae8 >2018-05-30 01:35:03,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:03,935-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:03,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:03,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:03,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:35:03,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5c87be8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:35:03,948-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5c87be8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:35:03,950-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5c87be8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5c87be8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:35:03,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 743c6b76 >2018-05-30 01:35:03,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:35:04,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:35:04,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:35:04,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 743c6b76 >2018-05-30 01:35:04,052-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:35:04,065-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [72a068e5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:35:04,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [72a068e5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 68f17b06 >2018-05-30 01:35:04,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [72a068e5] FINISH, SetVdsStatusVDSCommand, log id: 68f17b06 >2018-05-30 01:35:04,075-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72a068e5, Job ID: e9eba01a-082d-42ba-ab10-9d2186d0e72d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:35:04,079-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 401dd8f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:35:04,082-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [72a068e5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 401dd8f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:35:04,093-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5a03d551] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:35:04,094-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5a03d551] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49554884] Autorecovering 2 hosts >2018-05-30 01:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [49554884] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:40:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:40:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:40:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [110a6435] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 561d3177 >2018-05-30 01:40:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [110a6435] FINISH, SetVdsStatusVDSCommand, log id: 561d3177 >2018-05-30 01:40:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [110a6435] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:40:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [110a6435] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:40:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:40:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c97e39] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c6cc06c >2018-05-30 01:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c97e39] FINISH, SetVdsStatusVDSCommand, log id: 3c6cc06c >2018-05-30 01:40:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5c97e39] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 57033fa8 >2018-05-30 01:40:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5c97e39] FINISH, SetHaMaintenanceModeVDSCommand, log id: 57033fa8 >2018-05-30 01:40:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:40:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5c97e39] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:40:01,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1e6c3423] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 11777fe6 >2018-05-30 01:40:01,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [1e6c3423] FINISH, GetHardwareInfoVDSCommand, log id: 11777fe6 >2018-05-30 01:40:01,792-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7298f72b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:40:01,801-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [11f3449d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:40:01,806-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6aa07de1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:40:01,812-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6aa07de1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 105dc49 >2018-05-30 01:40:02,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:40:02,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6aa07de1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 105dc49 >2018-05-30 01:40:02,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:02,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:02,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:02,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:02,198-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6aa07de1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:02,201-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6aa07de1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:40:02,204-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6aa07de1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6aa07de1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:40:02,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 25a7220d >2018-05-30 01:40:02,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:40:02,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:40:02,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:40:02,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 25a7220d >2018-05-30 01:40:02,305-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:40:02,319-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2aa62886] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:40:02,320-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2aa62886] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dbf24a5 >2018-05-30 01:40:02,322-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2aa62886] FINISH, SetVdsStatusVDSCommand, log id: 3dbf24a5 >2018-05-30 01:40:02,353-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2aa62886, Job ID: 1bb9332c-4c5e-466f-87ad-f877d955f3ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:40:02,357-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 11f3449d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:40:02,360-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2aa62886] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 11f3449d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:40:02,371-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [24ed849a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:40:02,372-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [24ed849a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:40:02,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [615f82db] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43f45ab2 >2018-05-30 01:40:02,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [615f82db] FINISH, GetHardwareInfoVDSCommand, log id: 43f45ab2 >2018-05-30 01:40:02,991-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [5eb5c05c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:40:02,995-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [6b079332] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:40:03,001-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [165b596] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:40:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [165b596] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 789ba53 >2018-05-30 01:40:03,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:40:03,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [165b596] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 789ba53 >2018-05-30 01:40:03,392-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:03,393-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:03,397-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:03,398-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:40:03,402-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [165b596] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:40:03,406-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [165b596] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:40:03,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [165b596] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 165b596, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:40:03,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 550a346e >2018-05-30 01:40:03,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:40:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 550a346e >2018-05-30 01:40:03,510-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-32) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:40:03,523-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [43218d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:40:03,525-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [43218d6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 497e7a57 >2018-05-30 01:40:03,527-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [43218d6] FINISH, SetVdsStatusVDSCommand, log id: 497e7a57 >2018-05-30 01:40:03,532-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43218d6, Job ID: c7708b28-ee35-40ac-a2f9-821d81a9bf68, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:40:03,537-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6b079332, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:40:03,539-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [43218d6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6b079332, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:40:03,550-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7190c127] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:40:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7190c127] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [196c4617] Autorecovering 2 hosts >2018-05-30 01:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [196c4617] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:45:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76911a2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35981d44 >2018-05-30 01:45:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76911a2f] FINISH, SetVdsStatusVDSCommand, log id: 35981d44 >2018-05-30 01:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [76911a2f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:45:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [76911a2f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:45:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [608c7315] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 792d426 >2018-05-30 01:45:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [608c7315] FINISH, SetVdsStatusVDSCommand, log id: 792d426 >2018-05-30 01:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [608c7315] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7bd95008 >2018-05-30 01:45:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [608c7315] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7bd95008 >2018-05-30 01:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:45:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [608c7315] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:45:01,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4f0beda2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 359dd901 >2018-05-30 01:45:01,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [4f0beda2] FINISH, GetHardwareInfoVDSCommand, log id: 359dd901 >2018-05-30 01:45:01,254-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4c14d7e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:45:01,263-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [c4a66e3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:45:01,268-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3d8d6ce1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:45:01,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3d8d6ce1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 610bcf21 >2018-05-30 01:45:01,665-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:45:01,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3d8d6ce1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 610bcf21 >2018-05-30 01:45:01,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:01,670-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:01,674-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:01,675-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:01,679-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:01,680-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3d8d6ce1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:01,683-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3d8d6ce1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:45:01,685-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3d8d6ce1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3d8d6ce1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:45:01,693-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5978aba0 >2018-05-30 01:45:01,693-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:45:01,789-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:45:01,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:45:01,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:45:01,789-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:45:01,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5978aba0 >2018-05-30 01:45:01,790-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:45:01,803-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3678310e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:45:01,804-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3678310e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cba6366 >2018-05-30 01:45:01,806-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3678310e] FINISH, SetVdsStatusVDSCommand, log id: 4cba6366 >2018-05-30 01:45:01,839-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3678310e, Job ID: 1202d194-2297-4d00-a344-e27f605a883c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:45:01,844-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c4a66e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:45:01,846-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3678310e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c4a66e3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:45:01,857-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [f1e7622] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:45:01,858-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [f1e7622] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:45:03,191-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [16248d19] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4d24d770 >2018-05-30 01:45:03,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [16248d19] FINISH, GetHardwareInfoVDSCommand, log id: 4d24d770 >2018-05-30 01:45:03,207-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [537ae5a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:45:03,216-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [626b230e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:45:03,221-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [414da02a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:45:03,283-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [414da02a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15a68701 >2018-05-30 01:45:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:45:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [414da02a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 15a68701 >2018-05-30 01:45:03,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:03,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:03,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:03,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:03,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:45:03,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [414da02a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:45:03,668-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [414da02a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:45:03,670-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [414da02a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 414da02a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:45:03,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1a97ab5b >2018-05-30 01:45:03,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:45:03,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:45:03,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:45:03,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1a97ab5b >2018-05-30 01:45:03,775-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:45:03,789-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [18ba37db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:45:03,790-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18ba37db] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b7f42d5 >2018-05-30 01:45:03,792-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [18ba37db] FINISH, SetVdsStatusVDSCommand, log id: 4b7f42d5 >2018-05-30 01:45:03,825-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18ba37db, Job ID: 78932972-784a-467d-9516-5719c1258c66, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:45:03,829-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 626b230e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:45:03,832-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [18ba37db] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 626b230e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:45:03,842-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [c4e8613] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:45:03,843-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [c4e8613] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [44fb3106] Autorecovering 2 hosts >2018-05-30 01:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [44fb3106] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:50:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1f1c9e7a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c314a22 >2018-05-30 01:50:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [1f1c9e7a] FINISH, SetVdsStatusVDSCommand, log id: 7c314a22 >2018-05-30 01:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [1f1c9e7a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:50:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [1f1c9e7a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b981aff >2018-05-30 01:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] FINISH, SetVdsStatusVDSCommand, log id: 3b981aff >2018-05-30 01:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e290499 >2018-05-30 01:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5e2fa1f7] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e290499 >2018-05-30 01:50:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:50:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5e2fa1f7] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:50:01,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [74befe10] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2d3b6e8d >2018-05-30 01:50:01,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [74befe10] FINISH, GetHardwareInfoVDSCommand, log id: 2d3b6e8d >2018-05-30 01:50:01,551-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2b341f77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:50:01,555-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2d2cfb05] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:50:01,560-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [36e91a48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:50:01,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [36e91a48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e43664b >2018-05-30 01:50:01,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:50:01,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [36e91a48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e43664b >2018-05-30 01:50:01,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:01,941-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:01,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:01,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:01,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:01,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [36e91a48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:01,953-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [36e91a48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:50:01,956-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [36e91a48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36e91a48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:50:01,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 681c14f7 >2018-05-30 01:50:01,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:50:02,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:50:02,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:50:02,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 681c14f7 >2018-05-30 01:50:02,058-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:50:02,071-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2c730c95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:50:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c730c95] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32c5dbc1 >2018-05-30 01:50:02,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c730c95] FINISH, SetVdsStatusVDSCommand, log id: 32c5dbc1 >2018-05-30 01:50:02,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2c730c95, Job ID: db61b7b9-80ca-4d5e-b0ed-00973473394b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:50:02,085-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d2cfb05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:50:02,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c730c95] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d2cfb05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:50:02,098-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [5df8b247] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:50:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5df8b247] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:50:02,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7fe7aad7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4500993e >2018-05-30 01:50:02,456-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [7fe7aad7] FINISH, GetHardwareInfoVDSCommand, log id: 4500993e >2018-05-30 01:50:02,469-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [5d3fe84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:50:02,474-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [671949b3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:50:02,479-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2778b2d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:50:02,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2778b2d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d3b9210 >2018-05-30 01:50:02,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:50:02,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2778b2d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4d3b9210 >2018-05-30 01:50:02,868-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:02,869-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:02,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:02,873-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:02,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:50:02,878-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2778b2d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:50:02,881-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2778b2d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:50:02,883-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2778b2d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2778b2d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:50:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35b425a2 >2018-05-30 01:50:02,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:50:02,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-4) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:50:02,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:50:02,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:50:02,984-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:50:02,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-4) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35b425a2 >2018-05-30 01:50:02,985-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-4) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:50:02,997-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [413d5526] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:50:02,999-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [413d5526] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a02828e >2018-05-30 01:50:03,001-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [413d5526] FINISH, SetVdsStatusVDSCommand, log id: 1a02828e >2018-05-30 01:50:03,006-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 413d5526, Job ID: dd7e9b35-4dc3-4645-aed6-742301a43c30, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:50:03,011-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 671949b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:50:03,013-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413d5526] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 671949b3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:50:03,024-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [2709f439] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:50:03,025-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [2709f439] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:54:38,374-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [413a85b8] Backup check started. >2018-05-30 01:54:38,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [413a85b8] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: There is no full backup available, please run engine-backup to prevent data loss in case of corruption. >2018-05-30 01:54:38,387-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler7) [413a85b8] Backup check completed. >2018-05-30 01:54:43,461-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [7cc97a42] Attempting to update VMs/Templates Ovf. >2018-05-30 01:54:43,464-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 01:54:43,464-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 01:54:43,464-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:54:43,468-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 01:54:43,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 01:54:43,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 01:54:43,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 01:54:43,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 01:54:43,472-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 01:54:43,472-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6f29d6b1] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 01:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4b6bbb16] Autorecovering 2 hosts >2018-05-30 01:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [4b6bbb16] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:55:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:55:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:55:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [31d2ea05] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f99b734 >2018-05-30 01:55:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [31d2ea05] FINISH, SetVdsStatusVDSCommand, log id: f99b734 >2018-05-30 01:55:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [31d2ea05] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 01:55:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [31d2ea05] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:55:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 01:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [83b03b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5abac7a6 >2018-05-30 01:55:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [83b03b] FINISH, SetVdsStatusVDSCommand, log id: 5abac7a6 >2018-05-30 01:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [83b03b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52533a67 >2018-05-30 01:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [83b03b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52533a67 >2018-05-30 01:55:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 01:55:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [83b03b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 01:55:00,591-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [18f5cdc3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 34cd95bf >2018-05-30 01:55:00,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [18f5cdc3] FINISH, GetHardwareInfoVDSCommand, log id: 34cd95bf >2018-05-30 01:55:00,605-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [707551ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:55:00,614-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6691ea3a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:55:00,619-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [673912d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:55:00,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [673912d4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 68388e09 >2018-05-30 01:55:01,050-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:55:01,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [673912d4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 68388e09 >2018-05-30 01:55:01,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:01,055-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:01,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:01,060-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [673912d4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:01,067-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [673912d4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:55:01,069-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [673912d4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 673912d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:55:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 51a1a831 >2018-05-30 01:55:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:55:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:55:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:55:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 51a1a831 >2018-05-30 01:55:01,173-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:55:01,187-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [7a657a1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:55:01,189-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a657a1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3dca34bf >2018-05-30 01:55:01,191-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a657a1b] FINISH, SetVdsStatusVDSCommand, log id: 3dca34bf >2018-05-30 01:55:01,222-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a657a1b, Job ID: 292c5266-fed5-44e7-8780-c7e8ca33259e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:55:01,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6691ea3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:55:01,229-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a657a1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6691ea3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:55:01,240-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4f12e790] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 01:55:01,241-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4f12e790] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 01:55:02,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2709c7c1] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2ff78df3 >2018-05-30 01:55:02,958-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2709c7c1] FINISH, GetHardwareInfoVDSCommand, log id: 2ff78df3 >2018-05-30 01:55:02,971-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7c211f53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:55:02,975-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5797db61] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:55:02,980-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [53eb27f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 01:55:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [53eb27f2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55e9c6f0 >2018-05-30 01:55:04,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 01:55:04,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [53eb27f2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55e9c6f0 >2018-05-30 01:55:04,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:04,313-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:04,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:04,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:04,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 01:55:04,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [53eb27f2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 01:55:04,325-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [53eb27f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 01:55:04,328-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [53eb27f2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 53eb27f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 01:55:04,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 77f92c1b >2018-05-30 01:55:04,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:55:04,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 01:55:04,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 01:55:04,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 77f92c1b >2018-05-30 01:55:04,433-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-1) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 01:55:04,446-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [650af7c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:55:04,447-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [650af7c7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 748593ce >2018-05-30 01:55:04,449-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [650af7c7] FINISH, SetVdsStatusVDSCommand, log id: 748593ce >2018-05-30 01:55:04,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 650af7c7, Job ID: 9bab30b8-c806-49c9-82e3-c0bd44f2e35a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 01:55:04,462-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5797db61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 01:55:04,464-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [650af7c7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5797db61, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 01:55:04,475-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6ba333c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 01:55:04,476-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6ba333c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [70318fcf] Autorecovering 2 hosts >2018-05-30 02:00:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [70318fcf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:00:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d725ceb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 717e7d06 >2018-05-30 02:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d725ceb] FINISH, SetVdsStatusVDSCommand, log id: 717e7d06 >2018-05-30 02:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d725ceb] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:00:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [d725ceb] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4496989f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45a634df >2018-05-30 02:00:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4496989f] FINISH, SetVdsStatusVDSCommand, log id: 45a634df >2018-05-30 02:00:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4496989f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 62430700 >2018-05-30 02:00:00,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [4496989f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 62430700 >2018-05-30 02:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:00:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [4496989f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:00:01,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4f7cabaa] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 447822e7 >2018-05-30 02:00:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4f7cabaa] FINISH, GetHardwareInfoVDSCommand, log id: 447822e7 >2018-05-30 02:00:01,877-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [2f37cdf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:00:01,885-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [10ef15ab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:00:01,891-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [6f60e13e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:00:01,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [6f60e13e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24c8d114 >2018-05-30 02:00:02,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7faaa895] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c1ce797 >2018-05-30 02:00:02,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7faaa895] FINISH, GetHardwareInfoVDSCommand, log id: 1c1ce797 >2018-05-30 02:00:02,133-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6870dfff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:00:02,138-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1fe0be15] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:00:02,143-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7ae17ef0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:00:02,149-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7ae17ef0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5c374cd8 >2018-05-30 02:00:02,278-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:00:02,278-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [6f60e13e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24c8d114 >2018-05-30 02:00:02,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,283-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,290-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,295-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [6f60e13e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,298-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [6f60e13e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:00:02,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f60e13e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6f60e13e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:00:02,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4efaeabc >2018-05-30 02:00:02,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:00:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:00:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:00:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4efaeabc >2018-05-30 02:00:02,401-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:00:02,414-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [612850d7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:00:02,415-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [612850d7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ca6502a >2018-05-30 02:00:02,417-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [612850d7] FINISH, SetVdsStatusVDSCommand, log id: 3ca6502a >2018-05-30 02:00:02,450-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 612850d7, Job ID: 29fe176a-6cb1-4987-a507-3fd91e66a60f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:00:02,455-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10ef15ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:00:02,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [612850d7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10ef15ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:00:02,467-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2ec495e3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:00:02,468-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2ec495e3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:00:02,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:00:02,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7ae17ef0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5c374cd8 >2018-05-30 02:00:02,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,524-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,529-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:00:02,534-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7ae17ef0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:00:02,537-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7ae17ef0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:00:02,539-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ae17ef0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ae17ef0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:00:02,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 67b4f957 >2018-05-30 02:00:02,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:00:02,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:00:02,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:00:02,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 67b4f957 >2018-05-30 02:00:02,640-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:00:02,654-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [50cff606] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:00:02,655-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50cff606] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca77a27 >2018-05-30 02:00:02,657-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [50cff606] FINISH, SetVdsStatusVDSCommand, log id: 6ca77a27 >2018-05-30 02:00:02,663-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 50cff606, Job ID: e634146d-de21-4c0e-941e-5a58b79616d4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:00:02,667-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fe0be15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:00:02,669-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [50cff606] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fe0be15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:00:02,681-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5b62ad08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:00:02,682-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5b62ad08] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [553c0213] Autorecovering 2 hosts >2018-05-30 02:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [553c0213] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [63e97862] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fd7f5cd >2018-05-30 02:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [63e97862] FINISH, SetVdsStatusVDSCommand, log id: 1fd7f5cd >2018-05-30 02:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [63e97862] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:05:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [63e97862] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:05:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1c57127d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53c2b105 >2018-05-30 02:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1c57127d] FINISH, SetVdsStatusVDSCommand, log id: 53c2b105 >2018-05-30 02:05:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [1c57127d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 42938716 >2018-05-30 02:05:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [1c57127d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 42938716 >2018-05-30 02:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:05:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [1c57127d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:05:00,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [103cbc57] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7752a995 >2018-05-30 02:05:00,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [103cbc57] FINISH, GetHardwareInfoVDSCommand, log id: 7752a995 >2018-05-30 02:05:00,339-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [fcc8b3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:05:00,343-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [15bc10ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:05:00,348-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3265e880] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:05:00,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3265e880] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 74a0bc3 >2018-05-30 02:05:00,726-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:05:00,726-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [3265e880] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 74a0bc3 >2018-05-30 02:05:00,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:00,731-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:00,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:00,736-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:00,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:00,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [3265e880] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:00,743-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [3265e880] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:05:00,745-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3265e880] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3265e880, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:05:00,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1d2bed53 >2018-05-30 02:05:00,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:05:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:05:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:05:00,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1d2bed53 >2018-05-30 02:05:00,846-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:05:00,859-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [2ae27700] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:05:00,860-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ae27700] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33035567 >2018-05-30 02:05:00,863-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [2ae27700] FINISH, SetVdsStatusVDSCommand, log id: 33035567 >2018-05-30 02:05:00,868-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ae27700, Job ID: 6dfbca09-f0c4-4659-aa67-a07ebe2a559b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:05:00,873-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 15bc10ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:05:00,875-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2ae27700] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 15bc10ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:05:00,886-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [31def34f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:05:00,887-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [31def34f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:05:03,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6f97d848] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 62538582 >2018-05-30 02:05:03,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [6f97d848] FINISH, GetHardwareInfoVDSCommand, log id: 62538582 >2018-05-30 02:05:03,107-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [2453922c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:05:03,111-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [2417b762] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:05:03,116-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a3b63a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:05:03,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a3b63a9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16ea0cb1 >2018-05-30 02:05:03,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:05:03,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3a3b63a9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 16ea0cb1 >2018-05-30 02:05:03,509-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:03,509-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:03,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:03,514-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:05:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3a3b63a9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:05:03,522-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3a3b63a9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:05:03,524-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3a3b63a9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3a3b63a9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:05:03,527-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 399edc4f >2018-05-30 02:05:03,527-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:05:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:05:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:05:03,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 399edc4f >2018-05-30 02:05:03,627-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:05:03,640-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [6a17604a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:05:03,641-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a17604a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 725ac20a >2018-05-30 02:05:03,643-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [6a17604a] FINISH, SetVdsStatusVDSCommand, log id: 725ac20a >2018-05-30 02:05:03,649-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6a17604a, Job ID: 4b70c373-82ef-40f0-9605-ba5314191648, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:05:03,653-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2417b762, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:05:03,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6a17604a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2417b762, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:05:03,666-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [410def0d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:05:03,667-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [410def0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42939036] Autorecovering 2 hosts >2018-05-30 02:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [42939036] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21a4ec32] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f86c28a >2018-05-30 02:10:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [21a4ec32] FINISH, SetVdsStatusVDSCommand, log id: 6f86c28a >2018-05-30 02:10:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [21a4ec32] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [21a4ec32] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:10:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2ab960ed >2018-05-30 02:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] FINISH, SetVdsStatusVDSCommand, log id: 2ab960ed >2018-05-30 02:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12c1a927 >2018-05-30 02:10:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [7c8bfc8c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12c1a927 >2018-05-30 02:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:10:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [7c8bfc8c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:10:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56cdd912] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 37bd412e >2018-05-30 02:10:01,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56cdd912] FINISH, GetHardwareInfoVDSCommand, log id: 37bd412e >2018-05-30 02:10:01,277-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7fe9adc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:10:01,285-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [64cf5b00] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:10:01,290-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1d86b6e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:10:01,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1d86b6e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 48dc7ea2 >2018-05-30 02:10:01,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [728aaea3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 19ceb896 >2018-05-30 02:10:01,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [728aaea3] FINISH, GetHardwareInfoVDSCommand, log id: 19ceb896 >2018-05-30 02:10:01,510-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [4e4eaf28] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:10:01,514-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [384cba1d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:10:01,519-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6620d3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:10:01,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6620d3b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25ecd7ec >2018-05-30 02:10:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:10:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [1d86b6e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 48dc7ea2 >2018-05-30 02:10:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,690-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [1d86b6e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [1d86b6e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:10:01,695-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1d86b6e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1d86b6e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:10:01,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2555b5 >2018-05-30 02:10:01,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:10:01,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:10:01,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:10:01,798-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2555b5 >2018-05-30 02:10:01,798-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:10:01,811-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [43add784] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:10:01,812-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43add784] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 684b596a >2018-05-30 02:10:01,814-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [43add784] FINISH, SetVdsStatusVDSCommand, log id: 684b596a >2018-05-30 02:10:01,846-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 43add784, Job ID: 43c8e7e2-ca2e-424a-9fc9-4a4700c36b63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:10:01,851-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 64cf5b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:10:01,853-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43add784] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 64cf5b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:10:01,864-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [623434c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:10:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [623434c6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:10:01,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:10:01,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [6620d3b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25ecd7ec >2018-05-30 02:10:01,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,901-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:10:01,906-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [6620d3b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:10:01,908-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [6620d3b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:10:01,911-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6620d3b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6620d3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:10:01,914-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d8256f3 >2018-05-30 02:10:01,914-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:10:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:10:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:10:02,014-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d8256f3 >2018-05-30 02:10:02,014-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:10:02,027-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3d2eb2b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:10:02,028-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d2eb2b7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 61dcd41c >2018-05-30 02:10:02,031-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3d2eb2b7] FINISH, SetVdsStatusVDSCommand, log id: 61dcd41c >2018-05-30 02:10:02,036-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3d2eb2b7, Job ID: e903a2d3-d4f1-4434-b53b-cdcdfa78e9ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:10:02,041-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 384cba1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:10:02,043-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d2eb2b7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 384cba1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:10:02,053-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [3ebf9976] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:10:02,055-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3ebf9976] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4532c78d] Autorecovering 2 hosts >2018-05-30 02:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4532c78d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73920b12] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 175ac275 >2018-05-30 02:15:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73920b12] FINISH, SetVdsStatusVDSCommand, log id: 175ac275 >2018-05-30 02:15:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73920b12] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:15:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73920b12] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [259255be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21d20439 >2018-05-30 02:15:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [259255be] FINISH, SetVdsStatusVDSCommand, log id: 21d20439 >2018-05-30 02:15:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [259255be] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3ae52e0d >2018-05-30 02:15:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [259255be] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3ae52e0d >2018-05-30 02:15:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:15:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [259255be] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:15:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2d74167] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: e2d5f99 >2018-05-30 02:15:02,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2d74167] FINISH, GetHardwareInfoVDSCommand, log id: e2d5f99 >2018-05-30 02:15:02,531-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6fcc50db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:15:02,539-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1b863745] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:15:02,544-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [8485ed1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:15:02,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [8485ed1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e559967 >2018-05-30 02:15:02,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [723d919f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1c42f9a8 >2018-05-30 02:15:02,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [723d919f] FINISH, GetHardwareInfoVDSCommand, log id: 1c42f9a8 >2018-05-30 02:15:02,746-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [c2799bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:15:02,750-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [4b9e6904] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:15:02,756-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9dc0709] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:15:02,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9dc0709] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6711af5b >2018-05-30 02:15:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:15:02,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [8485ed1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e559967 >2018-05-30 02:15:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:02,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:02,944-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [8485ed1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:02,951-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [8485ed1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:15:02,954-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8485ed1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 8485ed1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:15:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41458143 >2018-05-30 02:15:02,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:15:03,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:15:03,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:15:03,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:15:03,059-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:15:03,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41458143 >2018-05-30 02:15:03,060-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:15:03,073-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6b3d0cf1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:15:03,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b3d0cf1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25a9e913 >2018-05-30 02:15:03,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6b3d0cf1] FINISH, SetVdsStatusVDSCommand, log id: 25a9e913 >2018-05-30 02:15:03,108-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6b3d0cf1, Job ID: 45ea22f6-caf8-41e7-8804-eaa857159ec8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:15:03,113-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1b863745, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:15:03,115-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6b3d0cf1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1b863745, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:15:03,127-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [215a5091] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:15:03,128-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [215a5091] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:15:05,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:15:05,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [9dc0709] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6711af5b >2018-05-30 02:15:05,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:05,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:05,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:05,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:05,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:15:05,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [9dc0709] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:15:05,094-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [9dc0709] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:15:05,097-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [9dc0709] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9dc0709, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:15:05,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 659fecd8 >2018-05-30 02:15:05,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:15:05,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:15:05,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:15:05,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 659fecd8 >2018-05-30 02:15:05,196-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:15:05,210-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1bd91343] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:15:05,211-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1bd91343] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b34edd8 >2018-05-30 02:15:05,214-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1bd91343] FINISH, SetVdsStatusVDSCommand, log id: 4b34edd8 >2018-05-30 02:15:05,220-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1bd91343, Job ID: c0a61f8e-1299-4bfe-89fe-1bf72020df3d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:15:05,225-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b9e6904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:15:05,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1bd91343] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b9e6904, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:15:05,238-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [1d9978ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:15:05,239-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1d9978ae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [369d31b8] Autorecovering 2 hosts >2018-05-30 02:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [369d31b8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:20:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73147984] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 638c3af0 >2018-05-30 02:20:00,051-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73147984] FINISH, SetVdsStatusVDSCommand, log id: 638c3af0 >2018-05-30 02:20:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73147984] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:20:00,054-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73147984] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:20:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4961693d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79223b6f >2018-05-30 02:20:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4961693d] FINISH, SetVdsStatusVDSCommand, log id: 79223b6f >2018-05-30 02:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4961693d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 39016ee7 >2018-05-30 02:20:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [4961693d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 39016ee7 >2018-05-30 02:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4961693d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:20:00,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4532c69a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1df41eed >2018-05-30 02:20:00,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4532c69a] FINISH, GetHardwareInfoVDSCommand, log id: 1df41eed >2018-05-30 02:20:00,725-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [7afa845a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:20:00,734-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [35fcecb7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:20:00,739-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [63505b9b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:20:00,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [63505b9b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 13b2bc52 >2018-05-30 02:20:01,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:20:01,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [63505b9b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 13b2bc52 >2018-05-30 02:20:01,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:01,190-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:01,194-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:01,195-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:01,199-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [63505b9b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:01,202-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [63505b9b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:20:01,205-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [63505b9b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63505b9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:20:01,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1e212bb9 >2018-05-30 02:20:01,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:20:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:20:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:20:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1e212bb9 >2018-05-30 02:20:01,309-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:20:01,322-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [36bc8106] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:20:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36bc8106] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20523b72 >2018-05-30 02:20:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36bc8106] FINISH, SetVdsStatusVDSCommand, log id: 20523b72 >2018-05-30 02:20:01,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36bc8106, Job ID: 6e1d8497-ce43-426f-acbf-42f4e296de15, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:20:01,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 35fcecb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:20:01,340-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36bc8106] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 35fcecb7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:20:01,352-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [78240ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:20:01,353-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [78240ef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:20:02,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2b28dbfe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 46b63e00 >2018-05-30 02:20:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [2b28dbfe] FINISH, GetHardwareInfoVDSCommand, log id: 46b63e00 >2018-05-30 02:20:02,918-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [46626a58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:20:02,922-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [4de074b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:20:02,928-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [11ecd4c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:20:02,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [11ecd4c6] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 648fae0e >2018-05-30 02:20:03,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:20:03,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [11ecd4c6] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 648fae0e >2018-05-30 02:20:03,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:03,309-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:03,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:03,314-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:03,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:20:03,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [11ecd4c6] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:20:03,321-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [11ecd4c6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:20:03,324-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [11ecd4c6] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 11ecd4c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:20:03,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d2a016d >2018-05-30 02:20:03,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:20:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:20:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:20:03,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d2a016d >2018-05-30 02:20:03,425-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:20:03,440-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1151005b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:20:03,441-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1151005b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 137524ae >2018-05-30 02:20:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1151005b] FINISH, SetVdsStatusVDSCommand, log id: 137524ae >2018-05-30 02:20:03,477-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1151005b, Job ID: 78395a27-8825-430e-90da-263fe8165db8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:20:03,482-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4de074b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:20:03,485-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1151005b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4de074b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:20:03,496-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7c83c5ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:20:03,497-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7c83c5ab] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7bd8dada] Autorecovering 2 hosts >2018-05-30 02:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7bd8dada] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:25:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5f78e91e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76743721 >2018-05-30 02:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5f78e91e] FINISH, SetVdsStatusVDSCommand, log id: 76743721 >2018-05-30 02:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5f78e91e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:25:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [5f78e91e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7403dddf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 25d48867 >2018-05-30 02:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7403dddf] FINISH, SetVdsStatusVDSCommand, log id: 25d48867 >2018-05-30 02:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [7403dddf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 52ec677c >2018-05-30 02:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [7403dddf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 52ec677c >2018-05-30 02:25:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7403dddf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:25:01,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [362fbf91] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 63ad69b1 >2018-05-30 02:25:01,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [362fbf91] FINISH, GetHardwareInfoVDSCommand, log id: 63ad69b1 >2018-05-30 02:25:01,149-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [73ebd26a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:25:01,158-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [646bb7f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:25:01,163-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7e32c74f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:25:01,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7e32c74f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22e27c95 >2018-05-30 02:25:01,548-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:25:01,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [7e32c74f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 22e27c95 >2018-05-30 02:25:01,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:01,553-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:01,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:01,557-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:01,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:01,562-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [7e32c74f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:01,564-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [7e32c74f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:25:01,567-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7e32c74f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7e32c74f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:25:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 43c1df7d >2018-05-30 02:25:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:25:01,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:25:01,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:25:01,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 43c1df7d >2018-05-30 02:25:01,673-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:25:01,686-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [54bcacff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:25:01,687-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [54bcacff] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35df8c69 >2018-05-30 02:25:01,689-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [54bcacff] FINISH, SetVdsStatusVDSCommand, log id: 35df8c69 >2018-05-30 02:25:01,722-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 54bcacff, Job ID: a476d01c-3af2-4d80-908a-3506eecfe317, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:25:01,727-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 646bb7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:25:01,729-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54bcacff] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 646bb7f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:25:01,741-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [1513378a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:25:01,742-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1513378a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:25:01,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12ebb2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6ed6a4a4 >2018-05-30 02:25:01,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [12ebb2] FINISH, GetHardwareInfoVDSCommand, log id: 6ed6a4a4 >2018-05-30 02:25:01,979-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [327b77bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:25:01,983-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [d3317a2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:25:01,988-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [74ff964f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:25:01,995-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [74ff964f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 100439c5 >2018-05-30 02:25:02,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:25:02,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [74ff964f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 100439c5 >2018-05-30 02:25:02,377-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:02,378-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:02,382-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:02,383-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:02,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:25:02,387-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [74ff964f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:25:02,390-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [74ff964f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:25:02,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [74ff964f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 74ff964f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:25:02,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a95d705 >2018-05-30 02:25:02,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:25:02,490-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a95d705 >2018-05-30 02:25:02,490-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:25:02,502-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [2fa787e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:25:02,503-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2fa787e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 315462f1 >2018-05-30 02:25:02,506-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [2fa787e3] FINISH, SetVdsStatusVDSCommand, log id: 315462f1 >2018-05-30 02:25:02,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2fa787e3, Job ID: 4c4b1dcd-d3b4-4430-8df8-e0de67ab1e4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:25:02,516-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d3317a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:25:02,518-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2fa787e3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d3317a2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:25:02,528-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [285a5b02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:25:02,529-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [285a5b02] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:30:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4b0e1fc8] Autorecovering 2 hosts >2018-05-30 02:30:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [4b0e1fc8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:30:00,011-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e161e18] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3effd5c3 >2018-05-30 02:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2e161e18] FINISH, SetVdsStatusVDSCommand, log id: 3effd5c3 >2018-05-30 02:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2e161e18] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:30:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [2e161e18] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:30:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [555fae6b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 12bc20ba >2018-05-30 02:30:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [555fae6b] FINISH, SetVdsStatusVDSCommand, log id: 12bc20ba >2018-05-30 02:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [555fae6b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 79e4e426 >2018-05-30 02:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [555fae6b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 79e4e426 >2018-05-30 02:30:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:30:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [555fae6b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:30:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56d875ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: eb6147f >2018-05-30 02:30:03,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [56d875ad] FINISH, GetHardwareInfoVDSCommand, log id: eb6147f >2018-05-30 02:30:03,175-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [323dadeb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:30:03,184-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [79529eb0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:30:03,189-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6294a7e7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:30:03,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6294a7e7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3443979e >2018-05-30 02:30:03,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:30:03,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [6294a7e7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3443979e >2018-05-30 02:30:03,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:03,587-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:03,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:03,591-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:03,595-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:03,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [6294a7e7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:03,599-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [6294a7e7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:30:03,601-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6294a7e7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6294a7e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:30:03,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 244bdd88 >2018-05-30 02:30:03,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:30:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:30:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:30:03,707-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 244bdd88 >2018-05-30 02:30:03,707-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:30:03,721-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [79bd7edb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:30:03,722-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [79bd7edb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52518b04 >2018-05-30 02:30:03,724-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [79bd7edb] FINISH, SetVdsStatusVDSCommand, log id: 52518b04 >2018-05-30 02:30:03,755-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 79bd7edb, Job ID: 30491a8b-4be3-4649-a593-3b1537ceabd1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:30:03,760-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 79529eb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:30:03,762-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [79bd7edb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 79529eb0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:30:03,773-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [66cf2d52] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:30:03,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [66cf2d52] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:30:03,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6a5b5f73] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5872be2e >2018-05-30 02:30:03,940-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6a5b5f73] FINISH, GetHardwareInfoVDSCommand, log id: 5872be2e >2018-05-30 02:30:03,954-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [7fed691a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:30:03,958-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [10feb5ab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:30:03,964-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [21a526e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:30:03,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [21a526e4] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59ac93f0 >2018-05-30 02:30:04,344-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:30:04,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [21a526e4] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 59ac93f0 >2018-05-30 02:30:04,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:04,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:04,352-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:04,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:04,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:30:04,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [21a526e4] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:30:04,361-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [21a526e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:30:04,364-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [21a526e4] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21a526e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:30:04,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 607f211a >2018-05-30 02:30:04,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:30:04,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:30:04,487-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:30:04,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:30:04,488-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:30:04,488-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 607f211a >2018-05-30 02:30:04,488-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:30:04,502-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3c319c50] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:30:04,503-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3c319c50] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2044d4f2 >2018-05-30 02:30:04,506-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3c319c50] FINISH, SetVdsStatusVDSCommand, log id: 2044d4f2 >2018-05-30 02:30:04,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c319c50, Job ID: b4545efa-cca4-4b79-ab2d-51e0fe6374a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:30:04,517-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 10feb5ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:30:04,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c319c50] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 10feb5ab, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:30:04,530-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [55fd9eaf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:30:04,531-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [55fd9eaf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d29498e] Autorecovering 2 hosts >2018-05-30 02:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6d29498e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:35:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:35:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:35:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f1b4558] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c5be0e >2018-05-30 02:35:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f1b4558] FINISH, SetVdsStatusVDSCommand, log id: 7c5be0e >2018-05-30 02:35:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f1b4558] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:35:00,048-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6f1b4558] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:35:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [42c88103] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2997975e >2018-05-30 02:35:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [42c88103] FINISH, SetVdsStatusVDSCommand, log id: 2997975e >2018-05-30 02:35:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [42c88103] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 65d7533f >2018-05-30 02:35:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [42c88103] FINISH, SetHaMaintenanceModeVDSCommand, log id: 65d7533f >2018-05-30 02:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [42c88103] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:35:01,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [512a0803] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4215d810 >2018-05-30 02:35:01,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [512a0803] FINISH, GetHardwareInfoVDSCommand, log id: 4215d810 >2018-05-30 02:35:01,444-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [47eed417] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:35:01,448-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [997b800] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:35:01,453-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [682c3c01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:35:01,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [682c3c01] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a01a68e >2018-05-30 02:35:01,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:35:01,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [682c3c01] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a01a68e >2018-05-30 02:35:01,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:01,844-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:01,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:01,848-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:01,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:01,853-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [682c3c01] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:01,855-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [682c3c01] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:35:01,858-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [682c3c01] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 682c3c01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:35:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5955ff18 >2018-05-30 02:35:01,865-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:35:01,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:35:01,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:35:01,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5955ff18 >2018-05-30 02:35:01,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:35:01,978-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [3f7d1313] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:35:01,979-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f7d1313] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b5bca2e >2018-05-30 02:35:01,982-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [3f7d1313] FINISH, SetVdsStatusVDSCommand, log id: 4b5bca2e >2018-05-30 02:35:01,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3f7d1313, Job ID: 75c8d9b1-9f6e-48b9-b599-807ed0f85ed4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:35:01,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 997b800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:35:01,996-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f7d1313] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 997b800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:35:02,007-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [1b437dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:35:02,008-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1b437dd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:35:02,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [22ac59da] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 18610499 >2018-05-30 02:35:02,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [22ac59da] FINISH, GetHardwareInfoVDSCommand, log id: 18610499 >2018-05-30 02:35:02,140-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [7ceeda5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:35:02,145-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [44f8acca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:35:02,150-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4d113ac2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:35:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4d113ac2] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 18f217a >2018-05-30 02:35:02,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:35:02,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4d113ac2] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 18f217a >2018-05-30 02:35:02,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:02,595-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:02,599-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:02,599-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:02,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:35:02,604-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4d113ac2] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:35:02,607-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4d113ac2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:35:02,609-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4d113ac2] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d113ac2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:35:02,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22a64c2f >2018-05-30 02:35:02,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:35:02,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:35:02,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:35:02,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22a64c2f >2018-05-30 02:35:02,715-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:35:02,729-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [4e0e7201] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:35:02,730-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4e0e7201] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4926bffb >2018-05-30 02:35:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4e0e7201] FINISH, SetVdsStatusVDSCommand, log id: 4926bffb >2018-05-30 02:35:02,741-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e0e7201, Job ID: d672bb8a-8b6c-4ac9-83a9-362dcad63cc2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:35:02,745-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 44f8acca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:35:02,748-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4e0e7201] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 44f8acca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:35:02,758-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [a6f9015] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:35:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [a6f9015] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfb874c] Autorecovering 2 hosts >2018-05-30 02:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7dfb874c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [36eac085] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 39eaf271 >2018-05-30 02:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [36eac085] FINISH, SetVdsStatusVDSCommand, log id: 39eaf271 >2018-05-30 02:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [36eac085] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:40:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [36eac085] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4052a812] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45a0041 >2018-05-30 02:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [4052a812] FINISH, SetVdsStatusVDSCommand, log id: 45a0041 >2018-05-30 02:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [4052a812] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 331167df >2018-05-30 02:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [4052a812] FINISH, SetHaMaintenanceModeVDSCommand, log id: 331167df >2018-05-30 02:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [4052a812] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:40:00,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6f16b5f9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6c4844b >2018-05-30 02:40:00,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [6f16b5f9] FINISH, GetHardwareInfoVDSCommand, log id: 6c4844b >2018-05-30 02:40:00,424-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3704ab0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:40:00,433-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [600b1ce7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:40:00,438-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [422fdd6c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:40:00,450-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [422fdd6c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e5f38b6 >2018-05-30 02:40:00,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:40:00,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [422fdd6c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7e5f38b6 >2018-05-30 02:40:00,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:00,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:00,835-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:00,839-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:00,840-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [422fdd6c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:00,843-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [422fdd6c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:40:00,846-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [422fdd6c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 422fdd6c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:40:00,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: bf5addf >2018-05-30 02:40:00,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:40:00,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: bf5addf >2018-05-30 02:40:00,947-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:40:00,962-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [15986d2f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:40:00,963-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [15986d2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b105066 >2018-05-30 02:40:00,965-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [15986d2f] FINISH, SetVdsStatusVDSCommand, log id: 2b105066 >2018-05-30 02:40:00,998-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 15986d2f, Job ID: 55546093-d4a6-4522-803a-507f3f0bff00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:40:01,003-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 600b1ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:40:01,006-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [15986d2f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 600b1ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:40:01,016-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [382e8f14] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:40:01,017-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [382e8f14] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:40:03,007-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [688d28cf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 71d74f12 >2018-05-30 02:40:03,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [688d28cf] FINISH, GetHardwareInfoVDSCommand, log id: 71d74f12 >2018-05-30 02:40:03,023-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [63108e39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:40:03,027-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [4be007f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:40:03,032-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5f317b7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:40:03,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5f317b7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f92d7f9 >2018-05-30 02:40:03,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:40:03,409-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [5f317b7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4f92d7f9 >2018-05-30 02:40:03,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:03,414-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:03,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:03,418-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:03,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:40:03,423-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [5f317b7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:40:03,426-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [5f317b7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:40:03,428-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5f317b7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5f317b7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:40:03,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a86b68 >2018-05-30 02:40:03,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:40:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:40:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:40:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a86b68 >2018-05-30 02:40:03,529-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:40:03,542-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [524f0417] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:40:03,544-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [524f0417] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f512a13 >2018-05-30 02:40:03,546-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [524f0417] FINISH, SetVdsStatusVDSCommand, log id: 3f512a13 >2018-05-30 02:40:03,551-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 524f0417, Job ID: feb96d6a-23c1-46a8-bfa5-cbbeeb34ad1e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:40:03,556-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4be007f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:40:03,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [524f0417] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4be007f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:40:03,569-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [36405e00] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:40:03,570-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [36405e00] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [761b134d] Autorecovering 2 hosts >2018-05-30 02:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [761b134d] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:45:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:45:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:45:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,052-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [21b5232e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43696b08 >2018-05-30 02:45:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [21b5232e] FINISH, SetVdsStatusVDSCommand, log id: 43696b08 >2018-05-30 02:45:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [21b5232e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:45:00,058-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [21b5232e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:45:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37095e8 >2018-05-30 02:45:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] FINISH, SetVdsStatusVDSCommand, log id: 37095e8 >2018-05-30 02:45:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6e18ff9b >2018-05-30 02:45:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [fe01a8e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6e18ff9b >2018-05-30 02:45:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:45:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [fe01a8e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:45:01,300-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [62221bff] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7d5f9b6f >2018-05-30 02:45:01,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [62221bff] FINISH, GetHardwareInfoVDSCommand, log id: 7d5f9b6f >2018-05-30 02:45:01,315-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5e179800] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:45:01,323-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1c2e26df] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:45:01,329-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [63c7df41] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:45:01,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [63c7df41] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 562e6e07 >2018-05-30 02:45:01,608-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4253d159] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 68dd1db4 >2018-05-30 02:45:01,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4253d159] FINISH, GetHardwareInfoVDSCommand, log id: 68dd1db4 >2018-05-30 02:45:01,623-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [59b86d2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:45:01,627-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [1637deeb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:45:01,632-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [592e8109] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:45:01,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [592e8109] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7635ea62 >2018-05-30 02:45:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:45:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [63c7df41] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 562e6e07 >2018-05-30 02:45:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:01,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:01,714-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:01,718-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:01,719-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [63c7df41] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:01,721-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [63c7df41] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:45:01,724-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63c7df41] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63c7df41, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:45:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2ef3c8fb >2018-05-30 02:45:01,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:45:01,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:45:01,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:45:01,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2ef3c8fb >2018-05-30 02:45:01,859-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:45:01,872-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [fdf8128] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:45:01,873-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [fdf8128] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 564394bc >2018-05-30 02:45:01,875-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [fdf8128] FINISH, SetVdsStatusVDSCommand, log id: 564394bc >2018-05-30 02:45:01,905-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: fdf8128, Job ID: 4cc9f5e4-049c-4bec-a516-fd7573902482, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:45:01,910-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1c2e26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:45:01,912-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fdf8128] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1c2e26df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:45:01,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [550b1a0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:45:01,924-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [550b1a0e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:45:02,013-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:45:02,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [592e8109] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7635ea62 >2018-05-30 02:45:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:02,018-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:02,022-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:02,023-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:02,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:45:02,027-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [592e8109] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:45:02,030-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [592e8109] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:45:02,033-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [592e8109] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 592e8109, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:45:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3abdb66 >2018-05-30 02:45:02,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:45:02,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:45:02,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:45:02,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3abdb66 >2018-05-30 02:45:02,134-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:45:02,147-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [40a7abd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:45:02,148-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [40a7abd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52814f78 >2018-05-30 02:45:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [40a7abd2] FINISH, SetVdsStatusVDSCommand, log id: 52814f78 >2018-05-30 02:45:02,156-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 40a7abd2, Job ID: b7f8cb60-25a4-416a-86ab-417e81fb414f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:45:02,161-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1637deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:45:02,163-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [40a7abd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1637deeb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:45:02,173-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [7dcc90b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:45:02,174-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7dcc90b8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df155d0] Autorecovering 2 hosts >2018-05-30 02:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [7df155d0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c64aaf7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e7f56b7 >2018-05-30 02:50:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [c64aaf7] FINISH, SetVdsStatusVDSCommand, log id: e7f56b7 >2018-05-30 02:50:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [c64aaf7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:50:00,081-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [c64aaf7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:50:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:50:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:50:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30267ffb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3408ea38 >2018-05-30 02:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [30267ffb] FINISH, SetVdsStatusVDSCommand, log id: 3408ea38 >2018-05-30 02:50:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30267ffb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6894063b >2018-05-30 02:50:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [30267ffb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6894063b >2018-05-30 02:50:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:50:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [30267ffb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:50:02,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54162563] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 52cb89f6 >2018-05-30 02:50:02,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54162563] FINISH, GetHardwareInfoVDSCommand, log id: 52cb89f6 >2018-05-30 02:50:02,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5e48d348] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 4fd8d6d >2018-05-30 02:50:02,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5e48d348] FINISH, GetHardwareInfoVDSCommand, log id: 4fd8d6d >2018-05-30 02:50:02,758-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [52321471] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:50:02,767-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4e49e17a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:50:02,772-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [43de80ca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:50:02,780-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [65df50ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:50:02,788-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [f1fdf12] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:50:02,794-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7f6bcefd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:50:02,838-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [43de80ca] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e90a999 >2018-05-30 02:50:02,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7f6bcefd] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79dbe611 >2018-05-30 02:50:03,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:50:03,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7f6bcefd] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79dbe611 >2018-05-30 02:50:03,232-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:50:03,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [43de80ca] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e90a999 >2018-05-30 02:50:03,236-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,236-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,243-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,244-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,245-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,248-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,249-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7f6bcefd] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:50:03,250-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [43de80ca] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:50:03,253-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7f6bcefd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:50:03,254-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [43de80ca] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:50:03,256-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f6bcefd] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f6bcefd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:50:03,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [43de80ca] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 43de80ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:50:03,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 41f60b7a >2018-05-30 02:50:03,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:50:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:50:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:50:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 41f60b7a >2018-05-30 02:50:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:50:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d81519c >2018-05-30 02:50:03,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:50:03,375-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [2b2755be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:50:03,376-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b2755be] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 402c250a >2018-05-30 02:50:03,378-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2b2755be] FINISH, SetVdsStatusVDSCommand, log id: 402c250a >2018-05-30 02:50:03,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2b2755be, Job ID: 9c4693ce-4366-4aa8-906b-d3af1cf170fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:50:03,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f1fdf12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:50:03,418-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2b2755be] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f1fdf12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:50:03,430-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [5adfeee3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:50:03,431-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [5adfeee3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:50:03,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:50:03,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:50:03,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:50:03,462-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:50:03,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d81519c >2018-05-30 02:50:03,462-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:50:03,476-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [3497e10d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:50:03,477-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3497e10d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6015d021 >2018-05-30 02:50:03,479-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3497e10d] FINISH, SetVdsStatusVDSCommand, log id: 6015d021 >2018-05-30 02:50:03,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3497e10d, Job ID: 8abb2902-fad5-452b-833d-72a9c45625e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:50:03,517-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4e49e17a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:50:03,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3497e10d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4e49e17a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:50:03,531-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [5471bbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:50:03,532-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [5471bbe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:54:43,472-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [2e9b92f3] Attempting to update VMs/Templates Ovf. >2018-05-30 02:54:43,475-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 02:54:43,475-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 02:54:43,476-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:54:43,480-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 02:54:43,483-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 02:54:43,483-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 02:54:43,483-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 02:54:43,483-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 02:54:43,484-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 02:54:43,484-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [324e7d97] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 02:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2e344ff3] Autorecovering 2 hosts >2018-05-30 02:55:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2e344ff3] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:55:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1f6340] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33264a07 >2018-05-30 02:55:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [e1f6340] FINISH, SetVdsStatusVDSCommand, log id: 33264a07 >2018-05-30 02:55:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [e1f6340] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 02:55:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [e1f6340] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:55:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 02:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4723f6d9 >2018-05-30 02:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] FINISH, SetVdsStatusVDSCommand, log id: 4723f6d9 >2018-05-30 02:55:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 22d033bd >2018-05-30 02:55:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [5409ed5b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 22d033bd >2018-05-30 02:55:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 02:55:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [5409ed5b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 02:55:01,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [ca53216] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 75b96997 >2018-05-30 02:55:01,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [ca53216] FINISH, GetHardwareInfoVDSCommand, log id: 75b96997 >2018-05-30 02:55:01,083-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [131cdeed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:55:01,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4060913b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: d047f2 >2018-05-30 02:55:01,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4060913b] FINISH, GetHardwareInfoVDSCommand, log id: d047f2 >2018-05-30 02:55:01,089-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [7108be4f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:55:01,095-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5668ac0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:55:01,101-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7a68c411] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:55:01,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5668ac0f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4e9f095c >2018-05-30 02:55:01,106-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [4826057b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:55:01,111-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5d2516ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 02:55:01,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5d2516ee] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f3a9398 >2018-05-30 02:55:01,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:55:01,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [5668ac0f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4e9f095c >2018-05-30 02:55:01,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,487-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,491-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 02:55:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5d2516ee] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6f3a9398 >2018-05-30 02:55:01,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,496-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [5668ac0f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,499-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [5668ac0f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:55:01,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,502-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5668ac0f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5668ac0f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:55:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75f164a3 >2018-05-30 02:55:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:55:01,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 02:55:01,508-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5d2516ee] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 02:55:01,511-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5d2516ee] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 02:55:01,513-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5d2516ee] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d2516ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:55:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:55:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:55:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75f164a3 >2018-05-30 02:55:01,602-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:55:01,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 377ca585 >2018-05-30 02:55:01,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 02:55:01,615-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [228d5db6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:55:01,616-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [228d5db6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62581e9e >2018-05-30 02:55:01,619-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [228d5db6] FINISH, SetVdsStatusVDSCommand, log id: 62581e9e >2018-05-30 02:55:01,624-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 228d5db6, Job ID: 410db48d-f282-408f-89a5-8b524f9ae6ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:55:01,629-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7108be4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:55:01,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [228d5db6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7108be4f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:55:01,641-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5a78f420] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 02:55:01,643-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5a78f420] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:55:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 02:55:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 02:55:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 377ca585 >2018-05-30 02:55:01,708-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 02:55:01,721-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [73a34712] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:55:01,722-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [73a34712] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d518416 >2018-05-30 02:55:01,724-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [73a34712] FINISH, SetVdsStatusVDSCommand, log id: 3d518416 >2018-05-30 02:55:01,729-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 73a34712, Job ID: 784fe7d7-04d2-4a78-9a7f-6eebb2c43b70, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 02:55:01,734-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4826057b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 02:55:01,736-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [73a34712] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4826057b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 02:55:01,746-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [41b332f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 02:55:01,747-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [41b332f8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:00:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1113e8ad] Autorecovering 2 hosts >2018-05-30 03:00:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [1113e8ad] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:00:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:00:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:00:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [545f3d61] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28a61d18 >2018-05-30 03:00:00,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [545f3d61] FINISH, SetVdsStatusVDSCommand, log id: 28a61d18 >2018-05-30 03:00:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [545f3d61] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:00:00,051-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [545f3d61] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:00:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:00:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:00:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [428a84ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7057c6cc >2018-05-30 03:00:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [428a84ed] FINISH, SetVdsStatusVDSCommand, log id: 7057c6cc >2018-05-30 03:00:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [428a84ed] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1b3520a3 >2018-05-30 03:00:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [428a84ed] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1b3520a3 >2018-05-30 03:00:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:00:00,076-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [428a84ed] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 428a84ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-30 03:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [428a84ed] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:00:02,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [65c348a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7c4cffc6 >2018-05-30 03:00:02,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [65c348a] FINISH, GetHardwareInfoVDSCommand, log id: 7c4cffc6 >2018-05-30 03:00:02,263-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [4ac64f9e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:00:02,268-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [75a7fab3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:00:02,273-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e73de89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:00:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e73de89] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2cdb70f1 >2018-05-30 03:00:02,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5cec2a64] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 12c8e62c >2018-05-30 03:00:02,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [5cec2a64] FINISH, GetHardwareInfoVDSCommand, log id: 12c8e62c >2018-05-30 03:00:02,508-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [7a2004f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:00:02,513-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [251412e8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:00:02,518-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [214c2296] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:00:02,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [214c2296] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 712bb4f0 >2018-05-30 03:00:02,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:00:02,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e73de89] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2cdb70f1 >2018-05-30 03:00:02,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,667-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,668-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,672-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e73de89] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,675-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e73de89] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:00:02,678-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e73de89] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e73de89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:00:02,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a1d1d8 >2018-05-30 03:00:02,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:00:02,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:00:02,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:00:02,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a1d1d8 >2018-05-30 03:00:02,779-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:00:02,793-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [f379df9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:00:02,794-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [f379df9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 796f9c62 >2018-05-30 03:00:02,797-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [f379df9] FINISH, SetVdsStatusVDSCommand, log id: 796f9c62 >2018-05-30 03:00:02,805-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f379df9, Job ID: d84721d3-550b-42af-9321-45c4cee8bda6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:00:02,810-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 75a7fab3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:00:02,812-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [f379df9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 75a7fab3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:00:02,823-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [50c8db07] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:00:02,824-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [50c8db07] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:00:02,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:00:02,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [214c2296] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 712bb4f0 >2018-05-30 03:00:02,895-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,896-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,900-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,900-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,904-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:00:02,905-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [214c2296] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:00:02,908-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [214c2296] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:00:02,911-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [214c2296] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 214c2296, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:00:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 47e5d083 >2018-05-30 03:00:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:00:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:00:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:00:03,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 47e5d083 >2018-05-30 03:00:03,011-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:00:03,023-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [186eb74b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:00:03,025-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [186eb74b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 17a57bcb >2018-05-30 03:00:03,027-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [186eb74b] FINISH, SetVdsStatusVDSCommand, log id: 17a57bcb >2018-05-30 03:00:03,032-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 186eb74b, Job ID: 324765ea-b2e9-46e8-bca0-fb22b7859cb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:00:03,037-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 251412e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:00:03,039-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [186eb74b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 251412e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:00:03,049-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [9beb6dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:00:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [9beb6dc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ad7e16] Autorecovering 2 hosts >2018-05-30 03:05:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [68ad7e16] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:05:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [34f1cf19] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2727ef33 >2018-05-30 03:05:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [34f1cf19] FINISH, SetVdsStatusVDSCommand, log id: 2727ef33 >2018-05-30 03:05:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,076-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [34f1cf19] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 34f1cf19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-30 03:05:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [34f1cf19] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:05:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [34f1cf19] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:05:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52cff895] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 35af292c >2018-05-30 03:05:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [52cff895] FINISH, SetVdsStatusVDSCommand, log id: 35af292c >2018-05-30 03:05:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [52cff895] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 971a761 >2018-05-30 03:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [52cff895] FINISH, SetHaMaintenanceModeVDSCommand, log id: 971a761 >2018-05-30 03:05:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:05:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [52cff895] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:05:00,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ca7dc28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 742a82de >2018-05-30 03:05:00,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5ca7dc28] FINISH, GetHardwareInfoVDSCommand, log id: 742a82de >2018-05-30 03:05:00,529-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5d511f2c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:05:00,537-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5b25e2b2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:05:00,542-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49881acf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:05:00,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49881acf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6a1833c3 >2018-05-30 03:05:00,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3be9dacf] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 17943e0b >2018-05-30 03:05:00,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [3be9dacf] FINISH, GetHardwareInfoVDSCommand, log id: 17943e0b >2018-05-30 03:05:00,665-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6d1f6564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:05:00,669-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [525f5094] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:05:00,675-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3bafa7ac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:05:00,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3bafa7ac] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 52715ec7 >2018-05-30 03:05:00,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:05:00,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49881acf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6a1833c3 >2018-05-30 03:05:00,940-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:00,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:00,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:00,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:00,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:00,950-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49881acf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:00,953-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49881acf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:05:00,955-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49881acf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49881acf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:05:00,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3ca70d6f >2018-05-30 03:05:00,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:05:01,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:05:01,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3bafa7ac] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 52715ec7 >2018-05-30 03:05:01,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:01,057-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:01,062-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:05:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:05:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:05:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3ca70d6f >2018-05-30 03:05:01,063-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:05:01,066-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:05:01,067-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3bafa7ac] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:05:01,071-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3bafa7ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:05:01,073-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3bafa7ac] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3bafa7ac, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:05:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6b14d575 >2018-05-30 03:05:01,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:05:01,078-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [63fe8db1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:05:01,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [63fe8db1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31e1156b >2018-05-30 03:05:01,082-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [63fe8db1] FINISH, SetVdsStatusVDSCommand, log id: 31e1156b >2018-05-30 03:05:01,115-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 63fe8db1, Job ID: 35ca4777-982a-416e-8250-b73482f796c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:05:01,120-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5b25e2b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:05:01,122-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe8db1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5b25e2b2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:05:01,133-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [76501e05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:05:01,134-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [76501e05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:05:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:05:01,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:05:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:05:01,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:05:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6b14d575 >2018-05-30 03:05:01,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:05:01,190-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [29b5cf2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:05:01,191-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [29b5cf2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50f796b8 >2018-05-30 03:05:01,193-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [29b5cf2b] FINISH, SetVdsStatusVDSCommand, log id: 50f796b8 >2018-05-30 03:05:01,199-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29b5cf2b, Job ID: b333ab19-10e1-4cd0-a2c7-028a6ea6ad9e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:05:01,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 525f5094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:05:01,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [29b5cf2b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 525f5094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:05:01,216-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [11f178b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:05:01,217-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [11f178b3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:10:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [264bf43c] Autorecovering 2 hosts >2018-05-30 03:10:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [264bf43c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:10:00,013-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:10:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:10:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,021-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1b972f6b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3033bfa0 >2018-05-30 03:10:00,023-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1b972f6b] FINISH, SetVdsStatusVDSCommand, log id: 3033bfa0 >2018-05-30 03:10:00,025-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,026-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1b972f6b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:10:00,026-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1b972f6b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:10:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:10:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:10:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d941db5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 13fc2b0b >2018-05-30 03:10:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1d941db5] FINISH, SetVdsStatusVDSCommand, log id: 13fc2b0b >2018-05-30 03:10:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1d941db5] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61289c8f >2018-05-30 03:10:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [1d941db5] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61289c8f >2018-05-30 03:10:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:10:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [1d941db5] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:10:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5f8c67a3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 45ec3c27 >2018-05-30 03:10:01,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5f8c67a3] FINISH, GetHardwareInfoVDSCommand, log id: 45ec3c27 >2018-05-30 03:10:01,772-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [2a3c314e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:10:01,777-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [7eca0e17] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:10:01,782-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3b2ea602] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:10:01,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3b2ea602] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6daa8e5d >2018-05-30 03:10:01,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [46140736] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1806e091 >2018-05-30 03:10:01,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [46140736] FINISH, GetHardwareInfoVDSCommand, log id: 1806e091 >2018-05-30 03:10:01,881-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [a2d3aa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:10:01,886-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [2f84a16b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:10:01,891-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [42b511f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:10:01,897-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [42b511f9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a491e41 >2018-05-30 03:10:02,162-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:10:02,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3b2ea602] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6daa8e5d >2018-05-30 03:10:02,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,167-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,170-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,171-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,176-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3b2ea602] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,179-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3b2ea602] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:10:02,182-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3b2ea602] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b2ea602, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:10:02,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 616345ac >2018-05-30 03:10:02,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:10:02,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:10:02,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [42b511f9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7a491e41 >2018-05-30 03:10:02,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,272-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,277-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:10:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:10:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:10:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: 616345ac >2018-05-30 03:10:02,281-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:10:02,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:10:02,283-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [42b511f9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:10:02,286-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [42b511f9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:10:02,289-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42b511f9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42b511f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:10:02,293-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6e16e967] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:10:02,294-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e16e967] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 252d4602 >2018-05-30 03:10:02,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3b7ebb04 >2018-05-30 03:10:02,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:10:02,297-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6e16e967] FINISH, SetVdsStatusVDSCommand, log id: 252d4602 >2018-05-30 03:10:02,329-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6e16e967, Job ID: ace5c1bd-b936-45b3-815f-3b08192c4f56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:10:02,333-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7eca0e17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:10:02,336-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6e16e967] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7eca0e17, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:10:02,347-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [25d8195c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:10:02,348-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [25d8195c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:10:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:10:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:10:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3b7ebb04 >2018-05-30 03:10:02,395-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:10:02,408-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [3e26e1f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:10:02,409-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3e26e1f1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 614016a6 >2018-05-30 03:10:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [3e26e1f1] FINISH, SetVdsStatusVDSCommand, log id: 614016a6 >2018-05-30 03:10:02,416-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3e26e1f1, Job ID: 6628effd-e09f-4072-88bc-dc8cd1a57db9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:10:02,421-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2f84a16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:10:02,423-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3e26e1f1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2f84a16b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:10:02,433-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [45f5e41a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:10:02,434-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [45f5e41a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3c31af0b] Autorecovering 2 hosts >2018-05-30 03:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3c31af0b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:15:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d6e1b9e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72487e39 >2018-05-30 03:15:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3d6e1b9e] FINISH, SetVdsStatusVDSCommand, log id: 72487e39 >2018-05-30 03:15:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3d6e1b9e] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:15:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3d6e1b9e] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:15:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5bcd9ba7 >2018-05-30 03:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] FINISH, SetVdsStatusVDSCommand, log id: 5bcd9ba7 >2018-05-30 03:15:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 24c6d7c >2018-05-30 03:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [6b3e6f4a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 24c6d7c >2018-05-30 03:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:15:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [6b3e6f4a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:15:02,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [21063390] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 17ca6825 >2018-05-30 03:15:02,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [21063390] FINISH, GetHardwareInfoVDSCommand, log id: 17ca6825 >2018-05-30 03:15:02,998-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [3fbe9149] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:15:03,007-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [6f2d8818] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:15:03,012-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3922b1a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:15:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3922b1a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: b278ae0 >2018-05-30 03:15:03,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [26729fb7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: c563eb3 >2018-05-30 03:15:03,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [26729fb7] FINISH, GetHardwareInfoVDSCommand, log id: c563eb3 >2018-05-30 03:15:03,115-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [796ad9d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:15:03,120-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [2c5d03de] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:15:03,125-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [732f3621] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:15:03,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [732f3621] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 376bf3ee >2018-05-30 03:15:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:15:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [3922b1a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: b278ae0 >2018-05-30 03:15:03,405-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,406-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,411-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,415-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [3922b1a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,418-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [3922b1a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:15:03,421-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3922b1a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3922b1a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:15:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e35c91 >2018-05-30 03:15:03,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:15:03,509-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:15:03,509-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [732f3621] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 376bf3ee >2018-05-30 03:15:03,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,514-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,518-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,519-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:15:03,523-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [732f3621] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:15:03,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:15:03,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:15:03,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:15:03,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:15:03,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e35c91 >2018-05-30 03:15:03,526-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:15:03,527-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [732f3621] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:15:03,530-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [732f3621] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 732f3621, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:15:03,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6d08a357 >2018-05-30 03:15:03,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:15:03,540-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [3373632] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:15:03,542-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3373632] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 513463d1 >2018-05-30 03:15:03,544-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3373632] FINISH, SetVdsStatusVDSCommand, log id: 513463d1 >2018-05-30 03:15:03,576-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3373632, Job ID: 685aa4a7-e503-4b43-8621-a8e961d039b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:15:03,581-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f2d8818, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:15:03,583-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3373632] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f2d8818, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:15:03,594-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [130ead5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:15:03,595-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [130ead5d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:15:03,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:15:03,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:15:03,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6d08a357 >2018-05-30 03:15:03,640-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:15:03,653-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [383c6363] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:15:03,654-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383c6363] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1cb3297 >2018-05-30 03:15:03,656-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [383c6363] FINISH, SetVdsStatusVDSCommand, log id: 1cb3297 >2018-05-30 03:15:03,662-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 383c6363, Job ID: 1b401707-df04-4f52-81b9-9eea1001a99b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:15:03,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2c5d03de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:15:03,669-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [383c6363] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2c5d03de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:15:03,679-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [6774119a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:15:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [6774119a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7ff116c1] Autorecovering 2 hosts >2018-05-30 03:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7ff116c1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:20:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51610388] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 425848a7 >2018-05-30 03:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [51610388] FINISH, SetVdsStatusVDSCommand, log id: 425848a7 >2018-05-30 03:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [51610388] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:20:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [51610388] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [66d089de] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1c0717f2 >2018-05-30 03:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [66d089de] FINISH, SetVdsStatusVDSCommand, log id: 1c0717f2 >2018-05-30 03:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [66d089de] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 308daf2b >2018-05-30 03:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [66d089de] FINISH, SetHaMaintenanceModeVDSCommand, log id: 308daf2b >2018-05-30 03:20:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:20:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [66d089de] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:20:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4a5ef427] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3136387b >2018-05-30 03:20:01,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4a5ef427] FINISH, GetHardwareInfoVDSCommand, log id: 3136387b >2018-05-30 03:20:01,241-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7aaaec3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:20:01,250-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [52eea52e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:20:01,255-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7423662] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:20:01,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7423662] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e82b976 >2018-05-30 03:20:01,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1ccc1792] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 735749d1 >2018-05-30 03:20:01,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1ccc1792] FINISH, GetHardwareInfoVDSCommand, log id: 735749d1 >2018-05-30 03:20:01,356-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [3a51f52d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:20:01,360-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [c51ce3e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:20:01,365-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2d9c22fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:20:01,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2d9c22fc] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 685d32c9 >2018-05-30 03:20:01,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:20:01,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7423662] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e82b976 >2018-05-30 03:20:01,657-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:01,658-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:01,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:01,663-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:01,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:01,667-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7423662] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:01,670-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7423662] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:20:01,673-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7423662] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7423662, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:20:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2b95fef2 >2018-05-30 03:20:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:20:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:20:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:20:01,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2b95fef2 >2018-05-30 03:20:01,773-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:20:01,787-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5c0184b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:20:01,788-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5c0184b0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 47aa3393 >2018-05-30 03:20:01,790-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5c0184b0] FINISH, SetVdsStatusVDSCommand, log id: 47aa3393 >2018-05-30 03:20:01,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c0184b0, Job ID: f7ab1000-0e8b-4ad8-ad1c-e380e047d87b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:20:01,826-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52eea52e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:20:01,828-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5c0184b0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52eea52e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:20:01,839-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [3ceca5f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:20:01,840-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [3ceca5f8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:20:02,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:20:02,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2d9c22fc] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 685d32c9 >2018-05-30 03:20:02,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:02,740-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:02,744-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:02,744-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:02,748-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:20:02,749-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [2d9c22fc] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:20:02,752-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2d9c22fc] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:20:02,755-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2d9c22fc] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2d9c22fc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:20:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 611aa3d9 >2018-05-30 03:20:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:20:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:20:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:20:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-34) [] FINISH, ConnectStoragePoolVDSCommand, log id: 611aa3d9 >2018-05-30 03:20:02,854-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:20:02,868-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [96645e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:20:02,869-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [96645e6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 16dbb2f8 >2018-05-30 03:20:02,871-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [96645e6] FINISH, SetVdsStatusVDSCommand, log id: 16dbb2f8 >2018-05-30 03:20:02,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 96645e6, Job ID: 87307ee1-474f-4ddc-a4a1-cffeb87828bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:20:02,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c51ce3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:20:02,884-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [96645e6] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c51ce3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:20:02,894-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [72c9c024] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:20:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [72c9c024] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:25:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [10bd92a2] Autorecovering 2 hosts >2018-05-30 03:25:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [10bd92a2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:25:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a978923] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e16731b >2018-05-30 03:25:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [3a978923] FINISH, SetVdsStatusVDSCommand, log id: 1e16731b >2018-05-30 03:25:00,048-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [3a978923] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:25:00,050-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [3a978923] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:25:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [296e4021] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 76a14a5f >2018-05-30 03:25:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [296e4021] FINISH, SetVdsStatusVDSCommand, log id: 76a14a5f >2018-05-30 03:25:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [296e4021] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1682ff99 >2018-05-30 03:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [296e4021] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1682ff99 >2018-05-30 03:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:25:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [296e4021] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:25:00,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5dd08b13] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 42979ad9 >2018-05-30 03:25:00,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [5dd08b13] FINISH, GetHardwareInfoVDSCommand, log id: 42979ad9 >2018-05-30 03:25:00,580-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7d208450] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:25:00,584-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [33df88a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:25:00,590-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49153462] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:25:00,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49153462] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1e05a186 >2018-05-30 03:25:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:25:01,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [49153462] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1e05a186 >2018-05-30 03:25:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:01,992-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [49153462] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:02,001-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [49153462] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:25:02,004-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [49153462] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 49153462, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:25:02,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15300b65 >2018-05-30 03:25:02,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:25:02,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:25:02,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:25:02,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15300b65 >2018-05-30 03:25:02,107-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:25:02,121-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [7898808b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:25:02,122-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7898808b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b91caf1 >2018-05-30 03:25:02,124-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7898808b] FINISH, SetVdsStatusVDSCommand, log id: b91caf1 >2018-05-30 03:25:02,133-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7898808b, Job ID: c41b7355-e751-4ee8-a161-4d584b8b8272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:25:02,138-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33df88a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:25:02,140-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7898808b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33df88a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:25:02,152-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [3153b12] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:25:02,153-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3153b12] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:25:02,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [666e3835] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7f30eeea >2018-05-30 03:25:02,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [666e3835] FINISH, GetHardwareInfoVDSCommand, log id: 7f30eeea >2018-05-30 03:25:02,452-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [18624a50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:25:02,457-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [36d848fe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:25:02,462-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [42012828] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:25:02,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [42012828] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 100244c7 >2018-05-30 03:25:02,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:25:02,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [42012828] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 100244c7 >2018-05-30 03:25:02,849-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:02,849-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:02,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:02,854-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:25:02,859-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [42012828] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:25:02,862-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [42012828] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:25:02,864-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [42012828] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 42012828, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:25:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 78634e16 >2018-05-30 03:25:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:25:02,963-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:25:02,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:25:02,963-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:25:02,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:25:02,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 78634e16 >2018-05-30 03:25:02,964-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:25:02,977-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [a753877] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:25:02,978-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a753877] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 574df750 >2018-05-30 03:25:02,980-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [a753877] FINISH, SetVdsStatusVDSCommand, log id: 574df750 >2018-05-30 03:25:02,986-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: a753877, Job ID: d39ecf41-f72e-4e87-afc8-35c08536e2f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:25:02,991-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36d848fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:25:02,993-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [a753877] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36d848fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:25:03,005-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6d93c232] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:25:03,007-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6d93c232] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [643f9577] Autorecovering 2 hosts >2018-05-30 03:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [643f9577] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:30:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8305d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33854b5f >2018-05-30 03:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3a8305d8] FINISH, SetVdsStatusVDSCommand, log id: 33854b5f >2018-05-30 03:30:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3a8305d8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:30:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [3a8305d8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [34f232d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e19fb89 >2018-05-30 03:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [34f232d4] FINISH, SetVdsStatusVDSCommand, log id: e19fb89 >2018-05-30 03:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [34f232d4] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: bbad2ea >2018-05-30 03:30:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [34f232d4] FINISH, SetHaMaintenanceModeVDSCommand, log id: bbad2ea >2018-05-30 03:30:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [34f232d4] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:30:00,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6fc7f2e4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5e93b5a1 >2018-05-30 03:30:00,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6fc7f2e4] FINISH, GetHardwareInfoVDSCommand, log id: 5e93b5a1 >2018-05-30 03:30:00,629-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1641f891] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:30:00,638-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [f35f845] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:30:00,643-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [348911f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:30:00,656-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [348911f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 157566bc >2018-05-30 03:30:01,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:30:01,038-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [348911f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 157566bc >2018-05-30 03:30:01,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:01,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:01,046-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:01,047-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:01,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:01,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [348911f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:01,054-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [348911f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:30:01,057-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [348911f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 348911f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:30:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 53e374fe >2018-05-30 03:30:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:30:01,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:30:01,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:30:01,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 53e374fe >2018-05-30 03:30:01,158-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:30:01,172-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [74de7a89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:30:01,173-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [74de7a89] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 460e046f >2018-05-30 03:30:01,175-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [74de7a89] FINISH, SetVdsStatusVDSCommand, log id: 460e046f >2018-05-30 03:30:01,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 74de7a89, Job ID: 7a417dac-8e4d-45e9-8843-0870c38bbe01, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:30:01,211-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: f35f845, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:30:01,213-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [74de7a89] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: f35f845, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:30:01,225-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4fcdcb4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:30:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4fcdcb4b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:30:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276ff507] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: b7539e3 >2018-05-30 03:30:03,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [276ff507] FINISH, GetHardwareInfoVDSCommand, log id: b7539e3 >2018-05-30 03:30:03,696-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [1de008aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:30:03,700-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [27690b83] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:30:03,705-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [35d96649] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:30:03,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [35d96649] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 73a677ab >2018-05-30 03:30:04,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:30:04,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [35d96649] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 73a677ab >2018-05-30 03:30:04,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:04,082-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:04,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:04,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:04,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:30:04,091-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [35d96649] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:30:04,094-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [35d96649] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:30:04,096-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [35d96649] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 35d96649, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:30:04,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2c32c61f >2018-05-30 03:30:04,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:30:04,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:30:04,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:30:04,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c32c61f >2018-05-30 03:30:04,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:30:04,210-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [1adcc379] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:30:04,211-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1adcc379] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31033401 >2018-05-30 03:30:04,213-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [1adcc379] FINISH, SetVdsStatusVDSCommand, log id: 31033401 >2018-05-30 03:30:04,219-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1adcc379, Job ID: 7a32be2b-3a80-4f88-bbda-0a4a4776294d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:30:04,223-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 27690b83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:30:04,226-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1adcc379] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 27690b83, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:30:04,236-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7c2517b1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:30:04,237-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7c2517b1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [68ab1ecd] Autorecovering 2 hosts >2018-05-30 03:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [68ab1ecd] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:35:00,030-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:35:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:35:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1f4f9064] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7630768 >2018-05-30 03:35:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1f4f9064] FINISH, SetVdsStatusVDSCommand, log id: 7630768 >2018-05-30 03:35:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1f4f9064] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:35:00,045-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [1f4f9064] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:35:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,058-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1fec2b55 >2018-05-30 03:35:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] FINISH, SetVdsStatusVDSCommand, log id: 1fec2b55 >2018-05-30 03:35:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4dc04ba8 >2018-05-30 03:35:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [1dbb105a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4dc04ba8 >2018-05-30 03:35:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:35:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [1dbb105a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:35:01,880-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [15e15557] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1f333b94 >2018-05-30 03:35:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [15e15557] FINISH, GetHardwareInfoVDSCommand, log id: 1f333b94 >2018-05-30 03:35:01,894-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [504880c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:35:01,903-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [198eab37] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:35:01,908-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2822fc33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:35:01,920-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2822fc33] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3af10a87 >2018-05-30 03:35:02,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [a774e28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4d3777a4 >2018-05-30 03:35:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [a774e28] FINISH, GetHardwareInfoVDSCommand, log id: 4d3777a4 >2018-05-30 03:35:02,148-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1a472ac0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:35:02,153-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [542a8ace] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:35:02,158-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2037b595] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:35:02,164-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2037b595] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1a946ef8 >2018-05-30 03:35:02,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:35:02,541-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [2037b595] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1a946ef8 >2018-05-30 03:35:02,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:02,545-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:02,549-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:02,550-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:02,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:02,555-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [2037b595] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:02,557-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [2037b595] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:35:02,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2037b595] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2037b595, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:35:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4638cf9b >2018-05-30 03:35:02,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:35:02,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:35:02,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:35:02,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4638cf9b >2018-05-30 03:35:02,663-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:35:02,677-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [18ae67d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:35:02,678-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18ae67d4] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73ce5622 >2018-05-30 03:35:02,681-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [18ae67d4] FINISH, SetVdsStatusVDSCommand, log id: 73ce5622 >2018-05-30 03:35:02,715-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18ae67d4, Job ID: b80cf50d-f814-4538-93f7-152a34202676, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:35:02,720-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 542a8ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:35:02,722-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [18ae67d4] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 542a8ace, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:35:02,732-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [6892754e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:35:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [6892754e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:35:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:35:03,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2822fc33] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3af10a87 >2018-05-30 03:35:03,242-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:03,243-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:03,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:03,247-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:03,251-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:35:03,252-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2822fc33] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:35:03,255-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2822fc33] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:35:03,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2822fc33] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2822fc33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:35:03,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 19dd29c8 >2018-05-30 03:35:03,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:35:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:35:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:35:03,360-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 19dd29c8 >2018-05-30 03:35:03,360-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:35:03,372-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [11999eab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:35:03,373-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [11999eab] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d790547 >2018-05-30 03:35:03,375-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [11999eab] FINISH, SetVdsStatusVDSCommand, log id: 2d790547 >2018-05-30 03:35:03,380-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 11999eab, Job ID: 665f4d45-f633-4e6e-9da8-4225a657efad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:35:03,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 198eab37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:35:03,387-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [11999eab] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 198eab37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:35:03,400-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [4feadf50] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:35:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [4feadf50] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:35:35,000-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler8) [78846500] Start deleteAgedOutAuditLogs >2018-05-30 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler6) [41d37a84] Start deleteAgedOutCommandEntities >2018-05-30 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler8) [78846500] Finished deleteAgedOutAuditLogs >2018-05-30 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler6) [41d37a84] Finished deleteAgedOutCommandEntities >2018-05-30 03:40:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5a0f002c] Autorecovering 2 hosts >2018-05-30 03:40:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [5a0f002c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:40:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [12cdc016] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f69088a >2018-05-30 03:40:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [12cdc016] FINISH, SetVdsStatusVDSCommand, log id: 4f69088a >2018-05-30 03:40:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [12cdc016] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:40:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [12cdc016] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:40:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4523baa6] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50c8c9fb >2018-05-30 03:40:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4523baa6] FINISH, SetVdsStatusVDSCommand, log id: 50c8c9fb >2018-05-30 03:40:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4523baa6] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5078870 >2018-05-30 03:40:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4523baa6] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5078870 >2018-05-30 03:40:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:40:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4523baa6] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:40:00,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4ec06d12] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1ea2586f >2018-05-30 03:40:00,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [4ec06d12] FINISH, GetHardwareInfoVDSCommand, log id: 1ea2586f >2018-05-30 03:40:00,294-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [42a69544] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:40:00,299-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [29746c75] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:40:00,304-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [9f865] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:40:00,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [9f865] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6682046b >2018-05-30 03:40:00,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:40:00,696-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [9f865] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6682046b >2018-05-30 03:40:00,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:00,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:00,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:00,705-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:00,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:00,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [9f865] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:00,712-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [9f865] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:40:00,714-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [9f865] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 9f865, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:40:00,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 584959c9 >2018-05-30 03:40:00,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:40:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:40:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:40:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 584959c9 >2018-05-30 03:40:00,818-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:40:00,831-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [4e4cd684] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:40:00,833-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4e4cd684] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7dd36fb2 >2018-05-30 03:40:00,835-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [4e4cd684] FINISH, SetVdsStatusVDSCommand, log id: 7dd36fb2 >2018-05-30 03:40:00,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4e4cd684, Job ID: 1ea7b0da-4d80-4d83-9b25-5e0ba4667343, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:40:00,847-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29746c75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:40:00,850-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e4cd684] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29746c75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:40:00,860-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6cad0f8c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:40:00,861-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6cad0f8c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:40:01,021-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6a584850] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 13d69abf >2018-05-30 03:40:01,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [6a584850] FINISH, GetHardwareInfoVDSCommand, log id: 13d69abf >2018-05-30 03:40:01,035-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [7613fe22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:40:01,039-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [40d72ee0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:40:01,044-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [75fd1f76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:40:01,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [75fd1f76] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 576c55a7 >2018-05-30 03:40:02,442-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:40:02,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [75fd1f76] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 576c55a7 >2018-05-30 03:40:02,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:02,447-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:02,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:02,451-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:02,455-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:40:02,456-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [75fd1f76] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:40:02,459-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [75fd1f76] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:40:02,461-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [75fd1f76] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75fd1f76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:40:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6f106980 >2018-05-30 03:40:02,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:40:02,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:40:02,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:40:02,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f106980 >2018-05-30 03:40:02,562-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:40:02,575-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [42b4ec83] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:40:02,576-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42b4ec83] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bbc9e51 >2018-05-30 03:40:02,577-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [42b4ec83] FINISH, SetVdsStatusVDSCommand, log id: 3bbc9e51 >2018-05-30 03:40:02,583-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 42b4ec83, Job ID: 9b517413-fd5e-41bd-9a1e-8ed6c64432e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:40:02,587-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 40d72ee0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:40:02,589-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42b4ec83] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 40d72ee0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:40:02,600-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [52553853] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:40:02,601-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [52553853] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4e42bd9a] Autorecovering 2 hosts >2018-05-30 03:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [4e42bd9a] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1e1b6226] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49aeabd5 >2018-05-30 03:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1e1b6226] FINISH, SetVdsStatusVDSCommand, log id: 49aeabd5 >2018-05-30 03:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [1e1b6226] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [1e1b6226] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [da56f4a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40b231a >2018-05-30 03:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [da56f4a] FINISH, SetVdsStatusVDSCommand, log id: 40b231a >2018-05-30 03:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [da56f4a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2e5e50f >2018-05-30 03:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [da56f4a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2e5e50f >2018-05-30 03:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [da56f4a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:45:00,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3192d626] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5e911f4d >2018-05-30 03:45:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3192d626] FINISH, GetHardwareInfoVDSCommand, log id: 5e911f4d >2018-05-30 03:45:00,364-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3026b0db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:45:00,372-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7081c77e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:45:00,377-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b93b87b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:45:00,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b93b87b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 48262122 >2018-05-30 03:45:00,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:45:00,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6b93b87b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 48262122 >2018-05-30 03:45:00,767-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:00,768-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:00,772-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:00,772-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:00,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:00,777-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6b93b87b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:00,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6b93b87b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:45:00,782-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b93b87b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6b93b87b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:45:00,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6a7e0389 >2018-05-30 03:45:00,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:45:00,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:45:00,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:45:00,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6a7e0389 >2018-05-30 03:45:00,886-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:45:00,898-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1ba1cbf5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:45:00,900-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1ba1cbf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b31837e >2018-05-30 03:45:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1ba1cbf5] FINISH, SetVdsStatusVDSCommand, log id: 3b31837e >2018-05-30 03:45:00,934-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1ba1cbf5, Job ID: 10da1865-ef13-4dab-b102-0dfa0d48a1ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:45:00,938-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7081c77e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:45:00,940-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1ba1cbf5] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7081c77e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:45:00,951-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [728cd0bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:45:00,952-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [728cd0bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:45:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [48d96b86] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 334801d2 >2018-05-30 03:45:01,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [48d96b86] FINISH, GetHardwareInfoVDSCommand, log id: 334801d2 >2018-05-30 03:45:01,721-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [2c54547e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:45:01,725-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [33c4bd1c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:45:01,730-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [59b3b2aa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:45:01,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [59b3b2aa] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f1ff772 >2018-05-30 03:45:02,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:45:02,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [59b3b2aa] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f1ff772 >2018-05-30 03:45:02,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:02,117-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:02,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:02,121-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:02,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:45:02,126-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [59b3b2aa] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:45:02,129-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [59b3b2aa] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:45:02,132-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [59b3b2aa] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 59b3b2aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:45:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3afabf40 >2018-05-30 03:45:02,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:45:02,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:45:02,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:45:02,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-22) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3afabf40 >2018-05-30 03:45:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-22) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:45:02,241-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [6908a261] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:45:02,242-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6908a261] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 52331419 >2018-05-30 03:45:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [6908a261] FINISH, SetVdsStatusVDSCommand, log id: 52331419 >2018-05-30 03:45:02,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6908a261, Job ID: 8068a67e-4f0f-434f-b9d8-394fecbb3ed8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:45:02,254-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33c4bd1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:45:02,256-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6908a261] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33c4bd1c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:45:02,267-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [71a365a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:45:02,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [71a365a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [315ad53f] Autorecovering 2 hosts >2018-05-30 03:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [315ad53f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:50:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:50:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:50:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,054-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,055-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36c189f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6eb00a9d >2018-05-30 03:50:00,058-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [36c189f0] FINISH, SetVdsStatusVDSCommand, log id: 6eb00a9d >2018-05-30 03:50:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [36c189f0] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:50:00,061-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [36c189f0] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [500ad926] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18829dac >2018-05-30 03:50:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [500ad926] FINISH, SetVdsStatusVDSCommand, log id: 18829dac >2018-05-30 03:50:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [500ad926] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1449ab7d >2018-05-30 03:50:00,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [500ad926] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1449ab7d >2018-05-30 03:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:50:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [500ad926] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:50:01,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7a075a3c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 599197fc >2018-05-30 03:50:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7a075a3c] FINISH, GetHardwareInfoVDSCommand, log id: 599197fc >2018-05-30 03:50:01,645-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [68045b3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:50:01,653-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [dc41994] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:50:01,658-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69d32300] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:50:01,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69d32300] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f906afb >2018-05-30 03:50:02,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d845e58] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 38095985 >2018-05-30 03:50:02,918-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7d845e58] FINISH, GetHardwareInfoVDSCommand, log id: 38095985 >2018-05-30 03:50:02,930-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [464524f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:50:02,935-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [21f7ca0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:50:02,939-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77a026f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:50:02,946-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77a026f7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1257571c >2018-05-30 03:50:03,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:50:03,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [69d32300] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7f906afb >2018-05-30 03:50:03,094-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,094-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,099-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,103-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [69d32300] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,106-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [69d32300] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:50:03,108-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [69d32300] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69d32300, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:50:03,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2231d526 >2018-05-30 03:50:03,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:50:03,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:50:03,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:50:03,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2231d526 >2018-05-30 03:50:03,213-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:50:03,225-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [7f0fcfb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:50:03,227-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7f0fcfb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 24fe278f >2018-05-30 03:50:03,229-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [7f0fcfb] FINISH, SetVdsStatusVDSCommand, log id: 24fe278f >2018-05-30 03:50:03,260-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f0fcfb, Job ID: 2fe1d799-0df1-4743-a426-f025df17bcf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:50:03,264-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: dc41994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:50:03,267-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7f0fcfb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: dc41994, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:50:03,277-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [76c2bed7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:50:03,278-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [76c2bed7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:50:03,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:50:03,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [77a026f7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1257571c >2018-05-30 03:50:03,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,337-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,341-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,341-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:50:03,346-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [77a026f7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:50:03,349-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [77a026f7] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:50:03,351-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [77a026f7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 77a026f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:50:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f9069c3 >2018-05-30 03:50:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:50:03,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:50:03,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:50:03,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: f9069c3 >2018-05-30 03:50:03,453-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:50:03,466-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [46ecfa1d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:50:03,467-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [46ecfa1d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26b28e78 >2018-05-30 03:50:03,469-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [46ecfa1d] FINISH, SetVdsStatusVDSCommand, log id: 26b28e78 >2018-05-30 03:50:03,474-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46ecfa1d, Job ID: 24910893-524c-4585-a6bc-6c97c321994d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:50:03,479-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 21f7ca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:50:03,481-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [46ecfa1d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 21f7ca0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:50:03,491-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [59a6eea6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:50:03,492-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [59a6eea6] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:54:43,484-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [3123f807] Attempting to update VMs/Templates Ovf. >2018-05-30 03:54:43,487-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 03:54:43,487-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 03:54:43,487-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:54:43,539-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 03:54:43,543-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 03:54:43,543-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 03:54:43,543-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 03:54:43,543-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 03:54:43,543-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 03:54:43,544-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [77cb6aa0] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 03:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [81e6294] Autorecovering 2 hosts >2018-05-30 03:55:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [81e6294] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:55:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [d0abe31] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c8ecdc5 >2018-05-30 03:55:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [d0abe31] FINISH, SetVdsStatusVDSCommand, log id: 6c8ecdc5 >2018-05-30 03:55:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [d0abe31] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 03:55:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [d0abe31] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:55:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:55:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 03:55:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [627a864b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 222e330a >2018-05-30 03:55:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [627a864b] FINISH, SetVdsStatusVDSCommand, log id: 222e330a >2018-05-30 03:55:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [627a864b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c6bed25 >2018-05-30 03:55:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [627a864b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c6bed25 >2018-05-30 03:55:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 03:55:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [627a864b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 03:55:00,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3aa26d3e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 179bf5a2 >2018-05-30 03:55:00,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3aa26d3e] FINISH, GetHardwareInfoVDSCommand, log id: 179bf5a2 >2018-05-30 03:55:01,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [151c78e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:55:01,009-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6738519e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:55:01,014-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [15a84530] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:55:01,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [15a84530] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1370a022 >2018-05-30 03:55:01,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2794b0b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 24479e05 >2018-05-30 03:55:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [2794b0b] FINISH, GetHardwareInfoVDSCommand, log id: 24479e05 >2018-05-30 03:55:01,076-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [675cd75e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:55:01,080-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [342d162b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:55:01,085-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [976f5d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 03:55:01,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [976f5d3] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 740ec081 >2018-05-30 03:55:01,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:55:01,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [15a84530] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1370a022 >2018-05-30 03:55:01,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,400-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,404-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,409-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [15a84530] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,412-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [15a84530] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:55:01,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [15a84530] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 15a84530, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:55:01,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 17bd5af9 >2018-05-30 03:55:01,422-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:55:01,462-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 03:55:01,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [976f5d3] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 740ec081 >2018-05-30 03:55:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,471-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 03:55:01,476-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [976f5d3] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 03:55:01,479-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [976f5d3] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 03:55:01,481-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [976f5d3] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 976f5d3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:55:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:55:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:55:01,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-31) [] FINISH, ConnectStoragePoolVDSCommand, log id: 17bd5af9 >2018-05-30 03:55:01,521-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-31) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:55:01,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7e713d86 >2018-05-30 03:55:01,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 03:55:01,535-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [14f71e8d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:55:01,536-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14f71e8d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 29822f1d >2018-05-30 03:55:01,538-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [14f71e8d] FINISH, SetVdsStatusVDSCommand, log id: 29822f1d >2018-05-30 03:55:01,569-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 14f71e8d, Job ID: 33f11979-0beb-4bd1-9066-f814ca0cb9e6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:55:01,573-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6738519e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:55:01,576-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14f71e8d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6738519e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:55:01,586-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [2ec9ecd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 03:55:01,587-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2ec9ecd] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:55:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 03:55:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 03:55:01,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7e713d86 >2018-05-30 03:55:01,623-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 03:55:01,636-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5f75cc1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:55:01,637-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5f75cc1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6406dae0 >2018-05-30 03:55:01,639-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5f75cc1c] FINISH, SetVdsStatusVDSCommand, log id: 6406dae0 >2018-05-30 03:55:01,645-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5f75cc1c, Job ID: ece4c200-8188-469c-a6f8-bc8a88d9c8fe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 03:55:01,649-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 342d162b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 03:55:01,651-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5f75cc1c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 342d162b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 03:55:01,662-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [8f2580d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 03:55:01,663-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [8f2580d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f2a0cdb] Autorecovering 2 hosts >2018-05-30 04:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [7f2a0cdb] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:00:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1a57c8a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 26257a80 >2018-05-30 04:00:00,079-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1a57c8a8] FINISH, SetVdsStatusVDSCommand, log id: 26257a80 >2018-05-30 04:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [1a57c8a8] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:00:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [1a57c8a8] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:00:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:00:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:00:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [32be9a51] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 769386b0 >2018-05-30 04:00:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [32be9a51] FINISH, SetVdsStatusVDSCommand, log id: 769386b0 >2018-05-30 04:00:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [32be9a51] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 697f96e9 >2018-05-30 04:00:00,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [32be9a51] FINISH, SetHaMaintenanceModeVDSCommand, log id: 697f96e9 >2018-05-30 04:00:00,107-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:00:00,108-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [32be9a51] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:00:02,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [16daab60] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 112fba8 >2018-05-30 04:00:02,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [16daab60] FINISH, GetHardwareInfoVDSCommand, log id: 112fba8 >2018-05-30 04:00:02,536-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f917b84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:00:02,544-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [6f98bad7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:00:02,549-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [672e3a48] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:00:02,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [672e3a48] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2389a7d2 >2018-05-30 04:00:02,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [48c1f028] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 57eec0d2 >2018-05-30 04:00:02,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [48c1f028] FINISH, GetHardwareInfoVDSCommand, log id: 57eec0d2 >2018-05-30 04:00:02,585-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [142b30a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:00:02,590-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [33a2973b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:00:02,594-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2706b3a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:00:02,600-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2706b3a0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 698a28c0 >2018-05-30 04:00:02,930-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:00:02,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [672e3a48] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2389a7d2 >2018-05-30 04:00:02,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,934-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,938-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,939-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,943-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [672e3a48] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,946-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [672e3a48] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:00:02,948-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [672e3a48] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 672e3a48, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:00:02,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2d29d792 >2018-05-30 04:00:02,954-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:00:02,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:00:02,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [2706b3a0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 698a28c0 >2018-05-30 04:00:02,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,978-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:00:02,987-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [2706b3a0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:00:02,989-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [2706b3a0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:00:02,992-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2706b3a0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2706b3a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:00:03,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:00:03,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:00:03,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-19) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2d29d792 >2018-05-30 04:00:03,054-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-19) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:00:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 28dfb1f0 >2018-05-30 04:00:03,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:00:03,068-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [57b6a322] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:00:03,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [57b6a322] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ad88ba1 >2018-05-30 04:00:03,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [57b6a322] FINISH, SetVdsStatusVDSCommand, log id: ad88ba1 >2018-05-30 04:00:03,103-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 57b6a322, Job ID: 4df62597-0245-4645-a23e-629d603428d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:00:03,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6f98bad7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:00:03,109-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b6a322] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6f98bad7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:00:03,120-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [449c40b4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:00:03,121-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [449c40b4] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:00:03,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:00:03,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:00:03,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 28dfb1f0 >2018-05-30 04:00:03,155-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:00:03,168-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [6444e040] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:00:03,169-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6444e040] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3049d267 >2018-05-30 04:00:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [6444e040] FINISH, SetVdsStatusVDSCommand, log id: 3049d267 >2018-05-30 04:00:03,176-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6444e040, Job ID: 6271370c-476e-40cd-b6e7-39549525dfc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:00:03,181-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 33a2973b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:00:03,183-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6444e040] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 33a2973b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:00:03,193-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [6fdc1260] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:00:03,194-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [6fdc1260] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:05:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2da9d06b] Autorecovering 2 hosts >2018-05-30 04:05:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [2da9d06b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:05:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52ba0bfd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 50732fed >2018-05-30 04:05:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [52ba0bfd] FINISH, SetVdsStatusVDSCommand, log id: 50732fed >2018-05-30 04:05:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [52ba0bfd] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:05:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [52ba0bfd] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:05:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1fb03873] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1888854 >2018-05-30 04:05:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [1fb03873] FINISH, SetVdsStatusVDSCommand, log id: 1888854 >2018-05-30 04:05:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1fb03873] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1afa3106 >2018-05-30 04:05:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [1fb03873] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1afa3106 >2018-05-30 04:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:05:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [1fb03873] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:05:00,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c507be8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 612b4bc0 >2018-05-30 04:05:00,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49c1d2ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38c722da >2018-05-30 04:05:00,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [49c1d2ad] FINISH, GetHardwareInfoVDSCommand, log id: 38c722da >2018-05-30 04:05:00,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3c507be8] FINISH, GetHardwareInfoVDSCommand, log id: 612b4bc0 >2018-05-30 04:05:00,808-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [621a1033] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:05:00,808-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4340a51e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:05:00,813-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [4f81415d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:05:00,813-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [e7429ee] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:05:00,819-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [393e9446] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:05:00,820-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6812468f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:05:00,832-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6812468f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3308e554 >2018-05-30 04:05:00,832-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [393e9446] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 30219bde >2018-05-30 04:05:01,209-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:05:01,209-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6812468f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3308e554 >2018-05-30 04:05:01,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,214-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:05:01,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [393e9446] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 30219bde >2018-05-30 04:05:01,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6812468f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,224-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,226-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6812468f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:05:01,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,229-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,229-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6812468f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6812468f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:05:01,232-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:05:01,233-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [393e9446] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:05:01,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 382608be >2018-05-30 04:05:01,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:05:01,237-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [393e9446] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:05:01,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [393e9446] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 393e9446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:05:01,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:05:01,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:05:01,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:05:01,336-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:05:01,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 382608be >2018-05-30 04:05:01,336-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:05:01,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74b2af8b >2018-05-30 04:05:01,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:05:01,350-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [13c685d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:05:01,351-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [13c685d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6f8753db >2018-05-30 04:05:01,353-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [13c685d3] FINISH, SetVdsStatusVDSCommand, log id: 6f8753db >2018-05-30 04:05:01,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 13c685d3, Job ID: db55e4df-f2fd-4c05-908e-d98883c7aeb1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:05:01,366-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4f81415d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:05:01,368-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [13c685d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4f81415d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:05:01,379-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [f8df71a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:05:01,380-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [f8df71a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:05:01,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:05:01,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:05:01,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74b2af8b >2018-05-30 04:05:01,441-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:05:01,453-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [462589dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:05:01,455-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [462589dc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41817421 >2018-05-30 04:05:01,457-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [462589dc] FINISH, SetVdsStatusVDSCommand, log id: 41817421 >2018-05-30 04:05:01,462-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 462589dc, Job ID: 6e31eba6-6421-4385-871b-8c8086770d18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:05:01,466-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: e7429ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:05:01,468-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [462589dc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: e7429ee, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:05:01,479-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [23f193c5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:05:01,480-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [23f193c5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4fbc0ad0] Autorecovering 2 hosts >2018-05-30 04:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4fbc0ad0] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:10:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55ad7fe7] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b0417ec >2018-05-30 04:10:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55ad7fe7] FINISH, SetVdsStatusVDSCommand, log id: 1b0417ec >2018-05-30 04:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55ad7fe7] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:10:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [55ad7fe7] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:10:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 656da39e >2018-05-30 04:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] FINISH, SetVdsStatusVDSCommand, log id: 656da39e >2018-05-30 04:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 99567f0 >2018-05-30 04:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [4f70d84f] FINISH, SetHaMaintenanceModeVDSCommand, log id: 99567f0 >2018-05-30 04:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:10:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [4f70d84f] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:10:02,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [50bc1f2] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43d71b54 >2018-05-30 04:10:02,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [50bc1f2] FINISH, GetHardwareInfoVDSCommand, log id: 43d71b54 >2018-05-30 04:10:02,154-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [426cf436] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:10:02,163-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [448770ad] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:10:02,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ad1c116] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 3dd00207 >2018-05-30 04:10:02,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ad1c116] FINISH, GetHardwareInfoVDSCommand, log id: 3dd00207 >2018-05-30 04:10:02,169-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7c579a2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:10:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7c579a2e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1102f6b7 >2018-05-30 04:10:02,202-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6846b247] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:10:02,210-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [2903212e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:10:02,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [137a4b8b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:10:02,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [137a4b8b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 11ee6065 >2018-05-30 04:10:02,564-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:10:02,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7c579a2e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1102f6b7 >2018-05-30 04:10:02,568-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,568-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,572-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,573-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,577-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,577-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7c579a2e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,580-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7c579a2e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:10:02,582-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7c579a2e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7c579a2e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:10:02,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 315153fc >2018-05-30 04:10:02,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:10:02,634-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:10:02,634-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [137a4b8b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 11ee6065 >2018-05-30 04:10:02,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,643-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,646-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:10:02,647-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [137a4b8b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:10:02,650-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [137a4b8b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:10:02,652-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [137a4b8b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 137a4b8b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:10:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:10:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:10:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:10:02,690-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:10:02,690-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-11) [] FINISH, ConnectStoragePoolVDSCommand, log id: 315153fc >2018-05-30 04:10:02,690-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-11) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:10:02,694-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 37f024a0 >2018-05-30 04:10:02,694-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:10:02,703-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [52c49a8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:10:02,704-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [52c49a8c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d448496 >2018-05-30 04:10:02,706-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [52c49a8c] FINISH, SetVdsStatusVDSCommand, log id: d448496 >2018-05-30 04:10:02,740-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 52c49a8c, Job ID: 34282f02-5e86-4d92-b0ff-6af24c10632b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:10:02,744-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 448770ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:10:02,746-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [52c49a8c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 448770ad, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:10:02,756-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [49e8078d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:10:02,757-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [49e8078d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:10:02,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:10:02,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:10:02,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 37f024a0 >2018-05-30 04:10:02,791-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:10:02,803-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [6fad4745] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:10:02,805-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6fad4745] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4d6ea658 >2018-05-30 04:10:02,807-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [6fad4745] FINISH, SetVdsStatusVDSCommand, log id: 4d6ea658 >2018-05-30 04:10:02,812-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 6fad4745, Job ID: 3e73e2a0-308a-4f63-88e9-3a5280c54174, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:10:02,816-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2903212e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:10:02,818-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6fad4745] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2903212e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:10:02,829-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [65f72f04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:10:02,830-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [65f72f04] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [708bf037] Autorecovering 2 hosts >2018-05-30 04:15:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [708bf037] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:15:00,015-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:15:00,021-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:15:00,021-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,021-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,023-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5a0a7e9b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11ca1de7 >2018-05-30 04:15:00,026-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [5a0a7e9b] FINISH, SetVdsStatusVDSCommand, log id: 11ca1de7 >2018-05-30 04:15:00,028-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,029-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [5a0a7e9b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:15:00,029-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5a0a7e9b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:15:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:15:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:15:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,041-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65214051] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e83708a >2018-05-30 04:15:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [65214051] FINISH, SetVdsStatusVDSCommand, log id: e83708a >2018-05-30 04:15:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [65214051] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 12495ee3 >2018-05-30 04:15:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [65214051] FINISH, SetHaMaintenanceModeVDSCommand, log id: 12495ee3 >2018-05-30 04:15:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:15:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [65214051] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:15:00,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [327d07af] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5bba3aae >2018-05-30 04:15:00,398-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [327d07af] FINISH, GetHardwareInfoVDSCommand, log id: 5bba3aae >2018-05-30 04:15:00,411-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [18026d64] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:15:00,419-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [b492edc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:15:00,424-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3ded4261] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:15:00,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3ded4261] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3bc609e7 >2018-05-30 04:15:00,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7b2b4093] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 60c2ddc7 >2018-05-30 04:15:00,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [7b2b4093] FINISH, GetHardwareInfoVDSCommand, log id: 60c2ddc7 >2018-05-30 04:15:00,498-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3b050204] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:15:00,502-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2bbfe039] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:15:00,508-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [783c8ff5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:15:00,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [783c8ff5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 64547ed8 >2018-05-30 04:15:00,816-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:15:00,816-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [3ded4261] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3bc609e7 >2018-05-30 04:15:00,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,821-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,825-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,829-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [3ded4261] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,833-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [3ded4261] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:15:00,835-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3ded4261] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3ded4261, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:15:00,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5ba74413 >2018-05-30 04:15:00,842-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:15:00,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:15:00,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [783c8ff5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 64547ed8 >2018-05-30 04:15:00,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,889-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,894-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:15:00,898-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [783c8ff5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:15:00,901-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [783c8ff5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:15:00,904-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [783c8ff5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 783c8ff5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:15:00,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:15:00,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:15:00,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ba74413 >2018-05-30 04:15:00,942-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:15:00,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5b2ce322 >2018-05-30 04:15:00,943-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:15:00,955-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [32086e53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:15:00,956-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [32086e53] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1639db2 >2018-05-30 04:15:00,959-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [32086e53] FINISH, SetVdsStatusVDSCommand, log id: 1639db2 >2018-05-30 04:15:00,964-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 32086e53, Job ID: 2ebbdecf-e8de-4cf9-9e66-6c060c73e165, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:15:00,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: b492edc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:15:00,971-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [32086e53] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: b492edc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:15:00,981-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [472dc71] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:15:00,982-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [472dc71] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:15:01,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:15:01,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:15:01,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-41) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5b2ce322 >2018-05-30 04:15:01,044-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-41) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:15:01,058-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [29c311ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:15:01,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [29c311ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 20609f6d >2018-05-30 04:15:01,061-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [29c311ed] FINISH, SetVdsStatusVDSCommand, log id: 20609f6d >2018-05-30 04:15:01,066-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29c311ed, Job ID: c36cc375-3508-4578-9d66-7a0a52377af0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:15:01,071-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bbfe039, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:15:01,073-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [29c311ed] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bbfe039, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:15:01,083-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [70c8ffc6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:15:01,084-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [70c8ffc6] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:20:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [37261055] Autorecovering 2 hosts >2018-05-30 04:20:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [37261055] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [702d3e47] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 756192df >2018-05-30 04:20:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [702d3e47] FINISH, SetVdsStatusVDSCommand, log id: 756192df >2018-05-30 04:20:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [702d3e47] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:20:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [702d3e47] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72387a8c >2018-05-30 04:20:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] FINISH, SetVdsStatusVDSCommand, log id: 72387a8c >2018-05-30 04:20:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1a70c229 >2018-05-30 04:20:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [5cc5b73d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1a70c229 >2018-05-30 04:20:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:20:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [5cc5b73d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:20:01,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54950fc6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6c79e364 >2018-05-30 04:20:01,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [54950fc6] FINISH, GetHardwareInfoVDSCommand, log id: 6c79e364 >2018-05-30 04:20:01,647-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [56f5f53f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:20:01,656-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5076460d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:20:01,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d238e42] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:20:01,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d238e42] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5450cdf0 >2018-05-30 04:20:01,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27d2151d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7fbe57ba >2018-05-30 04:20:01,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27d2151d] FINISH, GetHardwareInfoVDSCommand, log id: 7fbe57ba >2018-05-30 04:20:01,701-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [6c278d87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:20:01,706-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [475b8188] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:20:01,711-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1f22534a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:20:01,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1f22534a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26a6a718 >2018-05-30 04:20:02,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:20:02,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [6d238e42] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5450cdf0 >2018-05-30 04:20:02,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:02,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:02,068-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:02,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:02,072-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:02,073-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [6d238e42] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:02,076-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [6d238e42] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:20:02,079-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d238e42] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6d238e42, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:20:02,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 31cdf395 >2018-05-30 04:20:02,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:20:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:20:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:20:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-37) [] FINISH, ConnectStoragePoolVDSCommand, log id: 31cdf395 >2018-05-30 04:20:02,182-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-37) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:20:02,195-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [b513064] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:20:02,196-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b513064] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 62b65f75 >2018-05-30 04:20:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [b513064] FINISH, SetVdsStatusVDSCommand, log id: 62b65f75 >2018-05-30 04:20:02,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: b513064, Job ID: df47f7a6-0b11-44ad-a3d0-667499a8a838, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:20:02,236-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5076460d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:20:02,238-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [b513064] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5076460d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:20:02,249-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [bd8d490] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:20:02,250-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [bd8d490] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:20:03,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:20:03,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [1f22534a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26a6a718 >2018-05-30 04:20:03,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:03,092-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:03,096-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:03,097-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:03,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:20:03,101-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [1f22534a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:20:03,104-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [1f22534a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:20:03,107-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1f22534a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1f22534a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:20:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57a73ca4 >2018-05-30 04:20:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:20:03,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:20:03,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:20:03,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-44) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57a73ca4 >2018-05-30 04:20:03,206-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-44) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:20:03,219-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [4f4d468] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:20:03,221-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4f4d468] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 108b33ff >2018-05-30 04:20:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4f4d468] FINISH, SetVdsStatusVDSCommand, log id: 108b33ff >2018-05-30 04:20:03,228-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4f4d468, Job ID: d923d9b1-fcf1-4619-9f5b-170c270c69d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:20:03,233-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 475b8188, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:20:03,235-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [4f4d468] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 475b8188, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:20:03,246-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [44ffae5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:20:03,247-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [44ffae5e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [3bac3663] Autorecovering 2 hosts >2018-05-30 04:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [3bac3663] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:25:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6ca414ed] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79005363 >2018-05-30 04:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6ca414ed] FINISH, SetVdsStatusVDSCommand, log id: 79005363 >2018-05-30 04:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6ca414ed] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:25:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [6ca414ed] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:25:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:25:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:25:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [555a2afb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7ca3a8b6 >2018-05-30 04:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [555a2afb] FINISH, SetVdsStatusVDSCommand, log id: 7ca3a8b6 >2018-05-30 04:25:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [555a2afb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 790ec2ce >2018-05-30 04:25:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [555a2afb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 790ec2ce >2018-05-30 04:25:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [555a2afb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:25:00,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bcab9f6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 77d6f6 >2018-05-30 04:25:00,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3bcab9f6] FINISH, GetHardwareInfoVDSCommand, log id: 77d6f6 >2018-05-30 04:25:00,954-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2a8881f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:25:00,963-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [52ca6e4a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:25:00,968-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [48bded5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:25:00,979-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [48bded5f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24d4171f >2018-05-30 04:25:01,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:25:01,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [48bded5f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24d4171f >2018-05-30 04:25:01,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:01,354-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:01,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:01,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:01,362-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [48bded5f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:01,365-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [48bded5f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:25:01,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [48bded5f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 48bded5f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:25:01,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5eda525b >2018-05-30 04:25:01,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:25:01,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:25:01,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:25:01,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5eda525b >2018-05-30 04:25:01,475-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:25:01,489-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [12599ac1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:25:01,490-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [12599ac1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1edb3265 >2018-05-30 04:25:01,493-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [12599ac1] FINISH, SetVdsStatusVDSCommand, log id: 1edb3265 >2018-05-30 04:25:01,527-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 12599ac1, Job ID: 8ad014b6-f993-4776-aeaa-3df72582529e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:25:01,532-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52ca6e4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:25:01,534-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [12599ac1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52ca6e4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:25:01,545-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [ed36b0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:25:01,546-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [ed36b0a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:25:02,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [44883bce] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 798466f6 >2018-05-30 04:25:03,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [44883bce] FINISH, GetHardwareInfoVDSCommand, log id: 798466f6 >2018-05-30 04:25:03,012-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [f32c9d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:25:03,016-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5ffbc3c4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:25:03,021-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2f68b714] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:25:03,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2f68b714] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 246c0c20 >2018-05-30 04:25:03,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:25:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2f68b714] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 246c0c20 >2018-05-30 04:25:03,402-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:03,403-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:03,411-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:25:03,412-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2f68b714] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:25:03,415-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2f68b714] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:25:03,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2f68b714] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f68b714, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:25:03,420-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59ee69c8 >2018-05-30 04:25:03,420-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:25:03,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:25:03,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:25:03,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-12) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59ee69c8 >2018-05-30 04:25:03,515-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-12) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:25:03,527-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [29a4481b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:25:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29a4481b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 781f4662 >2018-05-30 04:25:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [29a4481b] FINISH, SetVdsStatusVDSCommand, log id: 781f4662 >2018-05-30 04:25:03,536-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 29a4481b, Job ID: 1dd2c3e3-6da4-4991-8af6-33d62a6dce55, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:25:03,540-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ffbc3c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:25:03,542-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [29a4481b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ffbc3c4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:25:03,554-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [83e707e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:25:03,555-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [83e707e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [8dbca0f] Autorecovering 2 hosts >2018-05-30 04:30:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [8dbca0f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ceede11] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74e25ede >2018-05-30 04:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [ceede11] FINISH, SetVdsStatusVDSCommand, log id: 74e25ede >2018-05-30 04:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [ceede11] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [ceede11] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:30:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:30:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,112-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [781ad20a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a6e09b9 >2018-05-30 04:30:00,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [781ad20a] FINISH, SetVdsStatusVDSCommand, log id: a6e09b9 >2018-05-30 04:30:00,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [781ad20a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 13929c83 >2018-05-30 04:30:00,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [781ad20a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 13929c83 >2018-05-30 04:30:00,119-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:30:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [781ad20a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:30:01,156-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33adfc71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 44bba2b4 >2018-05-30 04:30:01,157-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33adfc71] FINISH, GetHardwareInfoVDSCommand, log id: 44bba2b4 >2018-05-30 04:30:01,170-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4fb181a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:30:01,174-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2aab5a74] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:30:01,180-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62354ff5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:30:01,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62354ff5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 79d0636f >2018-05-30 04:30:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:30:01,570-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [62354ff5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 79d0636f >2018-05-30 04:30:01,574-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:01,575-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:01,579-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:01,579-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:01,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:01,584-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [62354ff5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:01,587-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [62354ff5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:30:01,590-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [62354ff5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 62354ff5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:30:01,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 70d6ce9c >2018-05-30 04:30:01,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:30:01,695-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:30:01,695-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:30:01,695-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 70d6ce9c >2018-05-30 04:30:01,695-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-38) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:30:01,709-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [26d2818d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:30:01,710-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [26d2818d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: bd021fe >2018-05-30 04:30:01,712-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [26d2818d] FINISH, SetVdsStatusVDSCommand, log id: bd021fe >2018-05-30 04:30:01,720-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 26d2818d, Job ID: b9fa3cdd-a100-46cd-8d3f-8d7b59427c91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:30:01,725-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2aab5a74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:30:01,727-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [26d2818d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2aab5a74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:30:01,737-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4e8e771] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:30:01,738-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4e8e771] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:30:02,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [590925d7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: ae79720 >2018-05-30 04:30:02,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [590925d7] FINISH, GetHardwareInfoVDSCommand, log id: ae79720 >2018-05-30 04:30:02,336-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3bf189ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:30:02,345-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [66c0d002] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:30:02,350-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5fd8bfbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:30:02,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5fd8bfbf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3e5bdebd >2018-05-30 04:30:04,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:30:04,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [5fd8bfbf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3e5bdebd >2018-05-30 04:30:04,684-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:04,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:04,688-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:04,689-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:04,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:30:04,694-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [5fd8bfbf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:30:04,696-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [5fd8bfbf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:30:04,699-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5fd8bfbf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5fd8bfbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:30:04,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 54e2e234 >2018-05-30 04:30:04,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:30:04,801-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:30:04,801-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:30:04,801-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 54e2e234 >2018-05-30 04:30:04,801-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-20) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:30:04,815-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [36e66a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:30:04,816-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [36e66a1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ab36235 >2018-05-30 04:30:04,818-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [36e66a1] FINISH, SetVdsStatusVDSCommand, log id: 5ab36235 >2018-05-30 04:30:04,850-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 36e66a1, Job ID: be4fe462-312d-45fe-873f-dff6397383fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:30:04,855-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 66c0d002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:30:04,857-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [36e66a1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 66c0d002, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:30:04,868-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [7da1b540] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:30:04,869-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7da1b540] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [184f095] Autorecovering 2 hosts >2018-05-30 04:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [184f095] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:35:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4903a70b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: a9f070 >2018-05-30 04:35:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [4903a70b] FINISH, SetVdsStatusVDSCommand, log id: a9f070 >2018-05-30 04:35:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [4903a70b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:35:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [4903a70b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:35:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:35:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:35:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59986535] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6968bbec >2018-05-30 04:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [59986535] FINISH, SetVdsStatusVDSCommand, log id: 6968bbec >2018-05-30 04:35:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [59986535] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2e637d0c >2018-05-30 04:35:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [59986535] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2e637d0c >2018-05-30 04:35:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:35:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [59986535] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:35:02,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f008328] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 43e806ba >2018-05-30 04:35:02,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3f008328] FINISH, GetHardwareInfoVDSCommand, log id: 43e806ba >2018-05-30 04:35:02,404-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [615cb22e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:35:02,413-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1fcac043] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:35:02,418-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4c47655a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:35:02,430-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4c47655a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 25332071 >2018-05-30 04:35:02,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [169b6782] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 367c4955 >2018-05-30 04:35:02,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [169b6782] FINISH, GetHardwareInfoVDSCommand, log id: 367c4955 >2018-05-30 04:35:02,532-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [3150491b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:35:02,537-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [110b08af] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:35:02,541-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169c2a24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:35:02,547-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169c2a24] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 56226c08 >2018-05-30 04:35:02,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:35:02,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [4c47655a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 25332071 >2018-05-30 04:35:02,812-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,813-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,818-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,821-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,822-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [4c47655a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,825-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [4c47655a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:35:02,827-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4c47655a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c47655a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:35:02,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 270df9bf >2018-05-30 04:35:02,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:35:02,913-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:35:02,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [169c2a24] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 56226c08 >2018-05-30 04:35:02,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,918-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,922-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,922-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,926-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:35:02,926-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [169c2a24] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:35:02,929-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [169c2a24] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:35:02,931-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [169c2a24] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 169c2a24, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:35:02,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:35:02,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:35:02,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 270df9bf >2018-05-30 04:35:02,934-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:35:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b030ba3 >2018-05-30 04:35:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:35:02,947-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [419e5b08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:35:02,948-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [419e5b08] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 768740c2 >2018-05-30 04:35:02,950-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [419e5b08] FINISH, SetVdsStatusVDSCommand, log id: 768740c2 >2018-05-30 04:35:02,983-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 419e5b08, Job ID: 520e086a-a409-4129-9153-17799cecb3a6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:35:02,987-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1fcac043, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:35:02,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [419e5b08] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1fcac043, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:35:03,000-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [10f9c677] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:35:03,001-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [10f9c677] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:35:03,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:35:03,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:35:03,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b030ba3 >2018-05-30 04:35:03,035-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-24) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:35:03,047-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [803795f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:35:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [803795f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 549f168f >2018-05-30 04:35:03,050-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [803795f] FINISH, SetVdsStatusVDSCommand, log id: 549f168f >2018-05-30 04:35:03,055-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 803795f, Job ID: 9df76752-927d-4e1f-9437-5021074f23e8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:35:03,060-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 110b08af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:35:03,061-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [803795f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 110b08af, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:35:03,071-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [43a9d28e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:35:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [43a9d28e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:40:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6a7e41f5] Autorecovering 2 hosts >2018-05-30 04:40:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [6a7e41f5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:40:00,030-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:40:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:40:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34f3eaf5] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45ddbab1 >2018-05-30 04:40:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34f3eaf5] FINISH, SetVdsStatusVDSCommand, log id: 45ddbab1 >2018-05-30 04:40:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34f3eaf5] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:40:00,044-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [34f3eaf5] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:40:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6da4218b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 32b35b71 >2018-05-30 04:40:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [6da4218b] FINISH, SetVdsStatusVDSCommand, log id: 32b35b71 >2018-05-30 04:40:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6da4218b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4936e6c1 >2018-05-30 04:40:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [6da4218b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4936e6c1 >2018-05-30 04:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:40:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [6da4218b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:40:00,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4c885a9b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4a448545 >2018-05-30 04:40:00,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [4c885a9b] FINISH, GetHardwareInfoVDSCommand, log id: 4a448545 >2018-05-30 04:40:00,600-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [1fe9f898] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:40:00,609-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [741b389c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:40:00,615-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [3365f2cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:40:00,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [3365f2cb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35192b8b >2018-05-30 04:40:00,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [55b13917] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 24b43c45 >2018-05-30 04:40:00,764-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [55b13917] FINISH, GetHardwareInfoVDSCommand, log id: 24b43c45 >2018-05-30 04:40:00,776-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7ab8690] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:40:00,780-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [6e210b9a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:40:00,785-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b3e7680] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:40:00,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b3e7680] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 45ca033c >2018-05-30 04:40:01,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:40:01,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [3365f2cb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 35192b8b >2018-05-30 04:40:01,009-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,010-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,014-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,015-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,019-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [3365f2cb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,022-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [3365f2cb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:40:01,025-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3365f2cb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3365f2cb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:40:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5a55e6f1 >2018-05-30 04:40:01,031-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:40:01,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:40:01,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:40:01,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5a55e6f1 >2018-05-30 04:40:01,125-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:40:01,138-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [714c20d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:40:01,139-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [714c20d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 634de580 >2018-05-30 04:40:01,142-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [714c20d] FINISH, SetVdsStatusVDSCommand, log id: 634de580 >2018-05-30 04:40:01,175-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 714c20d, Job ID: 5a5e4da1-aecd-4ab3-bd40-58d71d95a653, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:40:01,180-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 741b389c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:40:01,182-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [714c20d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 741b389c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:40:01,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:40:01,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5b3e7680] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 45ca033c >2018-05-30 04:40:01,188-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,189-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,193-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,195-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [1eaf9c5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:40:01,197-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [1eaf9c5f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:40:01,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:40:01,199-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [5b3e7680] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:40:01,251-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [5b3e7680] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:40:01,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5b3e7680] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5b3e7680, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:40:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35ff52e >2018-05-30 04:40:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:40:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:40:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:40:01,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-13) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35ff52e >2018-05-30 04:40:01,362-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-13) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:40:01,375-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [3c95ccd2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:40:01,376-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3c95ccd2] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54d7c7d8 >2018-05-30 04:40:01,378-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3c95ccd2] FINISH, SetVdsStatusVDSCommand, log id: 54d7c7d8 >2018-05-30 04:40:01,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3c95ccd2, Job ID: 9e659b23-a2a6-4a4b-8015-624a26701f9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:40:01,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6e210b9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:40:01,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c95ccd2] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6e210b9a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:40:01,428-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [178d8b4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:40:01,429-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [178d8b4b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1bb7d759] Autorecovering 2 hosts >2018-05-30 04:45:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [1bb7d759] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [75f48b1c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d47b525 >2018-05-30 04:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [75f48b1c] FINISH, SetVdsStatusVDSCommand, log id: d47b525 >2018-05-30 04:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [75f48b1c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:45:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [75f48b1c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [15094055] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2fee576a >2018-05-30 04:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [15094055] FINISH, SetVdsStatusVDSCommand, log id: 2fee576a >2018-05-30 04:45:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [15094055] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5c09eeb5 >2018-05-30 04:45:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [15094055] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5c09eeb5 >2018-05-30 04:45:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:45:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [15094055] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:45:01,833-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3837e18a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 416c44b8 >2018-05-30 04:45:01,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [3837e18a] FINISH, GetHardwareInfoVDSCommand, log id: 416c44b8 >2018-05-30 04:45:01,849-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [70a7ed83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:45:01,858-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [50873fb9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:45:01,863-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2107b584] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:45:01,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2107b584] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23818806 >2018-05-30 04:45:02,088-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4371efc0] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1ef2cd12 >2018-05-30 04:45:02,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [4371efc0] FINISH, GetHardwareInfoVDSCommand, log id: 1ef2cd12 >2018-05-30 04:45:02,104-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [60079d89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:45:02,109-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [138ea316] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:45:02,114-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a25153] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:45:02,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a25153] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6001bb62 >2018-05-30 04:45:02,254-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:45:02,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2107b584] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 23818806 >2018-05-30 04:45:02,258-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,259-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,263-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,263-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,268-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2107b584] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,271-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2107b584] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:45:02,273-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2107b584] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2107b584, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:45:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 793a0f45 >2018-05-30 04:45:02,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:45:02,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:45:02,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:45:02,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 793a0f45 >2018-05-30 04:45:02,384-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:45:02,398-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [51c09498] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:45:02,399-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [51c09498] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8017fa4 >2018-05-30 04:45:02,401-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [51c09498] FINISH, SetVdsStatusVDSCommand, log id: 8017fa4 >2018-05-30 04:45:02,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 51c09498, Job ID: 06d2b49d-79b1-46b8-beb1-452aa16ebb6a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:45:02,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 50873fb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:45:02,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [51c09498] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 50873fb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:45:02,428-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [7aa0d99e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:45:02,429-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7aa0d99e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:45:02,492-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:45:02,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a25153] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6001bb62 >2018-05-30 04:45:02,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,501-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:45:02,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a25153] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:45:02,509-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a25153] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:45:02,512-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a25153] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a25153, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:45:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4e2e29d8 >2018-05-30 04:45:02,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:45:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:45:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:45:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-48) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2e29d8 >2018-05-30 04:45:02,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-48) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:45:02,628-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [490fb97c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:45:02,629-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [490fb97c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ca3624e >2018-05-30 04:45:02,631-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [490fb97c] FINISH, SetVdsStatusVDSCommand, log id: 6ca3624e >2018-05-30 04:45:02,637-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 490fb97c, Job ID: 4edb7719-4583-485d-9246-4e651a68f416, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:45:02,641-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 138ea316, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:45:02,643-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [490fb97c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 138ea316, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:45:02,653-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [228f4d26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:45:02,654-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [228f4d26] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:50:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4cd6adb6] Autorecovering 2 hosts >2018-05-30 04:50:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [4cd6adb6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:50:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17dc4c37] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6164ae7c >2018-05-30 04:50:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [17dc4c37] FINISH, SetVdsStatusVDSCommand, log id: 6164ae7c >2018-05-30 04:50:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [17dc4c37] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:50:00,083-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [17dc4c37] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:50:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5721c550] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7796ba86 >2018-05-30 04:50:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5721c550] FINISH, SetVdsStatusVDSCommand, log id: 7796ba86 >2018-05-30 04:50:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5721c550] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 38be75a2 >2018-05-30 04:50:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [5721c550] FINISH, SetHaMaintenanceModeVDSCommand, log id: 38be75a2 >2018-05-30 04:50:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:50:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5721c550] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:50:00,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33ae8956] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 195f9b5d >2018-05-30 04:50:00,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [33ae8956] FINISH, GetHardwareInfoVDSCommand, log id: 195f9b5d >2018-05-30 04:50:00,351-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [2b447a48] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:50:00,359-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4774af54] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:50:00,365-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5298c26a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:50:00,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5298c26a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5e23b36f >2018-05-30 04:50:00,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:50:00,746-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [5298c26a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5e23b36f >2018-05-30 04:50:00,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:00,751-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:00,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:00,755-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:00,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:00,760-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [5298c26a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:00,763-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [5298c26a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:50:00,765-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5298c26a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5298c26a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:50:00,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a45660c >2018-05-30 04:50:00,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:50:00,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:50:00,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:50:00,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a45660c >2018-05-30 04:50:00,868-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:50:00,882-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [1d95b2af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:50:00,883-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d95b2af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1b20f12c >2018-05-30 04:50:00,885-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [1d95b2af] FINISH, SetVdsStatusVDSCommand, log id: 1b20f12c >2018-05-30 04:50:00,915-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1d95b2af, Job ID: 99c79e08-2182-4965-9040-22bb18303511, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:50:00,920-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4774af54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:50:00,922-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d95b2af] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4774af54, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:50:00,933-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [443eaf2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:50:00,934-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [443eaf2e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:50:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [78f29ea9] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 285697e8 >2018-05-30 04:50:03,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [78f29ea9] FINISH, GetHardwareInfoVDSCommand, log id: 285697e8 >2018-05-30 04:50:03,168-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [58d4986d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:50:03,172-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7acf7630] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:50:03,177-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6036e5be] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:50:03,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6036e5be] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7563918d >2018-05-30 04:50:03,563-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:50:03,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6036e5be] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7563918d >2018-05-30 04:50:03,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:03,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:03,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:03,572-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:03,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:50:03,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6036e5be] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:50:03,579-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6036e5be] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:50:03,581-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6036e5be] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6036e5be, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:50:03,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2264770b >2018-05-30 04:50:03,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:50:03,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:50:03,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:50:03,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2264770b >2018-05-30 04:50:03,682-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:50:03,695-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [243013a9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:50:03,696-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [243013a9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31d5641e >2018-05-30 04:50:03,698-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [243013a9] FINISH, SetVdsStatusVDSCommand, log id: 31d5641e >2018-05-30 04:50:03,703-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 243013a9, Job ID: 54fc9c71-ba3a-4a3f-8b93-80484f0eba74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:50:03,708-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7acf7630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:50:03,710-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [243013a9] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7acf7630, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:50:03,720-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4a255bef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:50:03,721-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4a255bef] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:54:43,544-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [3c038fa3] Attempting to update VMs/Templates Ovf. >2018-05-30 04:54:43,547-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 04:54:43,547-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 04:54:43,547-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:54:43,552-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 04:54:43,555-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 04:54:43,555-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 04:54:43,555-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 04:54:43,555-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 04:54:43,556-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 04:54:43,556-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [5cb12576] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 04:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [71b42b9c] Autorecovering 2 hosts >2018-05-30 04:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [71b42b9c] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:55:00,011-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:55:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:55:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,019-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69e706c1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 759ce807 >2018-05-30 04:55:00,021-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [69e706c1] FINISH, SetVdsStatusVDSCommand, log id: 759ce807 >2018-05-30 04:55:00,022-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,024-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [69e706c1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 04:55:00,024-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [69e706c1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:55:00,029-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:55:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 04:55:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,035-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,037-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [13053773] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 134b8098 >2018-05-30 04:55:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [13053773] FINISH, SetVdsStatusVDSCommand, log id: 134b8098 >2018-05-30 04:55:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [13053773] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51b4347a >2018-05-30 04:55:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [13053773] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51b4347a >2018-05-30 04:55:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 04:55:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [13053773] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 04:55:01,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [53c4f6df] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 7308c50c >2018-05-30 04:55:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [53c4f6df] FINISH, GetHardwareInfoVDSCommand, log id: 7308c50c >2018-05-30 04:55:01,339-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [62e8195a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:55:01,344-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [55274243] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:55:01,349-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2ddb9072] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:55:01,355-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2ddb9072] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 443f0476 >2018-05-30 04:55:01,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7c1d3122] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 438d336e >2018-05-30 04:55:01,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [7c1d3122] FINISH, GetHardwareInfoVDSCommand, log id: 438d336e >2018-05-30 04:55:01,571-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [605a8305] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:55:01,576-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [363de69e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:55:01,581-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7bd033ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 04:55:01,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7bd033ba] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5616d124 >2018-05-30 04:55:01,738-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:55:01,738-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2ddb9072] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 443f0476 >2018-05-30 04:55:01,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,743-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,746-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,747-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,751-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,752-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2ddb9072] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,754-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2ddb9072] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:55:01,757-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ddb9072] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2ddb9072, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:55:01,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 59fb118 >2018-05-30 04:55:01,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:55:01,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:55:01,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:55:01,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 59fb118 >2018-05-30 04:55:01,856-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:55:01,869-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [56784bbd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:55:01,870-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [56784bbd] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 15b0232b >2018-05-30 04:55:01,872-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [56784bbd] FINISH, SetVdsStatusVDSCommand, log id: 15b0232b >2018-05-30 04:55:01,878-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56784bbd, Job ID: db76f258-a79d-4842-a1d5-3b11b84a1236, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:55:01,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 55274243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:55:01,884-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [56784bbd] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 55274243, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:55:01,896-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [53e755fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 04:55:01,897-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [53e755fe] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 04:55:01,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 04:55:01,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [7bd033ba] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5616d124 >2018-05-30 04:55:01,957-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,957-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,962-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 04:55:01,966-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [7bd033ba] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 04:55:01,969-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [7bd033ba] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 04:55:01,972-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7bd033ba] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7bd033ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 04:55:01,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2da86659 >2018-05-30 04:55:01,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 04:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 04:55:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2da86659 >2018-05-30 04:55:02,073-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 04:55:02,086-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [72643ff7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:55:02,087-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72643ff7] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: e97bbe6 >2018-05-30 04:55:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [72643ff7] FINISH, SetVdsStatusVDSCommand, log id: e97bbe6 >2018-05-30 04:55:02,095-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72643ff7, Job ID: f56b61ea-6914-4fdf-b370-ffa9b96d5a56, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 04:55:02,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 363de69e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 04:55:02,102-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [72643ff7] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 363de69e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 04:55:02,112-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [31290e4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 04:55:02,113-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [31290e4e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1e6bcfd6] Autorecovering 2 hosts >2018-05-30 05:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [1e6bcfd6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:00:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:00:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:00:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3a0b167f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69bc1645 >2018-05-30 05:00:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3a0b167f] FINISH, SetVdsStatusVDSCommand, log id: 69bc1645 >2018-05-30 05:00:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3a0b167f] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:00:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3a0b167f] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:00:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [126a0b87] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c38e610 >2018-05-30 05:00:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [126a0b87] FINISH, SetVdsStatusVDSCommand, log id: 2c38e610 >2018-05-30 05:00:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [126a0b87] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6f87dde3 >2018-05-30 05:00:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [126a0b87] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6f87dde3 >2018-05-30 05:00:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:00:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [126a0b87] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:00:02,612-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [19d45516] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6cab0603 >2018-05-30 05:00:02,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [19d45516] FINISH, GetHardwareInfoVDSCommand, log id: 6cab0603 >2018-05-30 05:00:02,628-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5d3ff604] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:00:02,637-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [63a31020] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:00:02,642-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [107226] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:00:02,653-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [107226] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c460ed6 >2018-05-30 05:00:02,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7a4d2cb8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 38fd44be >2018-05-30 05:00:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [7a4d2cb8] FINISH, GetHardwareInfoVDSCommand, log id: 38fd44be >2018-05-30 05:00:02,778-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [585433f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:00:02,782-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [649a25d5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:00:02,787-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1bbe3637] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:00:02,794-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1bbe3637] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 321543a8 >2018-05-30 05:00:03,034-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:00:03,034-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [107226] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7c460ed6 >2018-05-30 05:00:03,038-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:03,039-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:03,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:03,043-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:03,048-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [107226] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:03,051-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [107226] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:00:03,053-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [107226] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 107226, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:00:03,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 347d25e6 >2018-05-30 05:00:03,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:00:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:00:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:00:03,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 347d25e6 >2018-05-30 05:00:03,154-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:00:03,167-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [414bfe4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:00:03,168-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [414bfe4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10f71153 >2018-05-30 05:00:03,171-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [414bfe4b] FINISH, SetVdsStatusVDSCommand, log id: 10f71153 >2018-05-30 05:00:03,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 414bfe4b, Job ID: 0aef0a8c-2d7a-4e77-b288-c25169f3318d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:00:03,208-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 63a31020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:00:03,210-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [414bfe4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 63a31020, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:00:03,221-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3f53093c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:00:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3f53093c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:00:05,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:00:05,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [1bbe3637] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 321543a8 >2018-05-30 05:00:05,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:05,107-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:05,111-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:05,112-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:05,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:00:05,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [1bbe3637] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:00:05,119-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [1bbe3637] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:00:05,122-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1bbe3637] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1bbe3637, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:00:05,124-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 55f3ce19 >2018-05-30 05:00:05,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:00:05,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:00:05,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:00:05,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-21) [] FINISH, ConnectStoragePoolVDSCommand, log id: 55f3ce19 >2018-05-30 05:00:05,221-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-21) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:00:05,250-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [58fa78d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:00:05,251-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [58fa78d8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f96cfea >2018-05-30 05:00:05,254-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [58fa78d8] FINISH, SetVdsStatusVDSCommand, log id: 3f96cfea >2018-05-30 05:00:05,285-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 58fa78d8, Job ID: cfef6249-7b4a-4c3c-9da4-5864d57efd27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:00:05,290-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 649a25d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:00:05,292-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58fa78d8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 649a25d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:00:05,304-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7e5eec53] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:00:05,305-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7e5eec53] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:05:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [61b8e18e] Autorecovering 2 hosts >2018-05-30 05:05:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [61b8e18e] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:05:00,010-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:05:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:05:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,017-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,018-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [35feac78] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 63b7c32e >2018-05-30 05:05:00,020-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [35feac78] FINISH, SetVdsStatusVDSCommand, log id: 63b7c32e >2018-05-30 05:05:00,021-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,023-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [35feac78] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:05:00,023-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [35feac78] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:05:00,028-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:05:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:05:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,035-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10b4aded] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2f898978 >2018-05-30 05:05:00,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [10b4aded] FINISH, SetVdsStatusVDSCommand, log id: 2f898978 >2018-05-30 05:05:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [10b4aded] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 21674136 >2018-05-30 05:05:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [10b4aded] FINISH, SetHaMaintenanceModeVDSCommand, log id: 21674136 >2018-05-30 05:05:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:05:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [10b4aded] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:05:00,764-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4af8be81] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 46bf3569 >2018-05-30 05:05:00,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4af8be81] FINISH, GetHardwareInfoVDSCommand, log id: 46bf3569 >2018-05-30 05:05:00,778-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [25ed4f4c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:05:00,786-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [78c02b95] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:05:00,791-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4574092] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:05:00,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4574092] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 72272262 >2018-05-30 05:05:01,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:05:01,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [4574092] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 72272262 >2018-05-30 05:05:01,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:01,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:01,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:01,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:01,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:01,200-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [4574092] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:01,203-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [4574092] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:05:01,206-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4574092] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4574092, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:05:01,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7b151251 >2018-05-30 05:05:01,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:05:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7b151251 >2018-05-30 05:05:01,312-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:05:01,325-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [3723b44] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:05:01,326-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3723b44] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b5a1b1d >2018-05-30 05:05:01,329-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3723b44] FINISH, SetVdsStatusVDSCommand, log id: 4b5a1b1d >2018-05-30 05:05:01,360-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 3723b44, Job ID: 6aade07b-ef82-4bbe-99cd-359c425e2b1d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:05:01,365-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 78c02b95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:05:01,367-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3723b44] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 78c02b95, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:05:01,378-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [3635a245] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:05:01,379-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [3635a245] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:05:03,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68e2db28] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 52d3b119 >2018-05-30 05:05:03,116-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [68e2db28] FINISH, GetHardwareInfoVDSCommand, log id: 52d3b119 >2018-05-30 05:05:03,129-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [72a5e226] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:05:03,133-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1be2ed6e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:05:03,138-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [23bb10d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:05:03,145-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [23bb10d8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: da3f1a8 >2018-05-30 05:05:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:05:03,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [23bb10d8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: da3f1a8 >2018-05-30 05:05:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:03,516-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:03,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:05:03,525-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [23bb10d8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:05:03,528-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [23bb10d8] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:05:03,531-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [23bb10d8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 23bb10d8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:05:03,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3216fc80 >2018-05-30 05:05:03,533-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:05:03,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:05:03,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:05:03,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3216fc80 >2018-05-30 05:05:03,632-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:05:03,646-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [7a311a4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:05:03,647-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a311a4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4aa8be43 >2018-05-30 05:05:03,649-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7a311a4d] FINISH, SetVdsStatusVDSCommand, log id: 4aa8be43 >2018-05-30 05:05:03,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7a311a4d, Job ID: a7fa7511-46df-43ca-8808-35074a2dcb43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:05:03,659-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1be2ed6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:05:03,662-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [7a311a4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1be2ed6e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:05:03,672-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [4825ce17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:05:03,673-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4825ce17] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [44a18af8] Autorecovering 2 hosts >2018-05-30 05:10:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [44a18af8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:10:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [176441ae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7bddcbb9 >2018-05-30 05:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [176441ae] FINISH, SetVdsStatusVDSCommand, log id: 7bddcbb9 >2018-05-30 05:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [176441ae] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:10:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [176441ae] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:10:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7161c4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cf9fd32 >2018-05-30 05:10:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7161c4d] FINISH, SetVdsStatusVDSCommand, log id: 4cf9fd32 >2018-05-30 05:10:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7161c4d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 45b0fa9f >2018-05-30 05:10:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [7161c4d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 45b0fa9f >2018-05-30 05:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:10:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [7161c4d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:10:01,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3dd2908b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76731c8 >2018-05-30 05:10:01,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [3dd2908b] FINISH, GetHardwareInfoVDSCommand, log id: 76731c8 >2018-05-30 05:10:01,364-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7cb58f3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:10:01,368-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [6fe16ffe] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:10:01,373-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5e64d031] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:10:01,385-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5e64d031] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4ca0886c >2018-05-30 05:10:01,760-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:10:01,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5e64d031] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4ca0886c >2018-05-30 05:10:01,764-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:01,765-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:01,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:01,770-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:01,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:01,774-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5e64d031] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:01,777-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5e64d031] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:10:01,779-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5e64d031] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5e64d031, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:10:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7fe10a19 >2018-05-30 05:10:01,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:10:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:10:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:10:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7fe10a19 >2018-05-30 05:10:01,882-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:10:01,895-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [7f33dfec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:10:01,896-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f33dfec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 469c7699 >2018-05-30 05:10:01,898-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [7f33dfec] FINISH, SetVdsStatusVDSCommand, log id: 469c7699 >2018-05-30 05:10:01,906-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7f33dfec, Job ID: 35bdb91c-fba2-4f2b-b551-ea80369cd2d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:10:01,910-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6fe16ffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:10:01,912-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7f33dfec] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6fe16ffe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:10:01,923-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [768c77ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:10:01,924-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [768c77ad] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:10:01,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4838742c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 3bee49df >2018-05-30 05:10:01,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [4838742c] FINISH, GetHardwareInfoVDSCommand, log id: 3bee49df >2018-05-30 05:10:01,960-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [447dfdf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:10:01,964-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [29eddee7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:10:01,969-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [58620569] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:10:01,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [58620569] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12b32aeb >2018-05-30 05:10:02,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:10:02,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [58620569] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 12b32aeb >2018-05-30 05:10:02,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:02,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:02,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:02,354-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:02,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:10:02,359-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [58620569] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:10:02,361-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [58620569] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:10:02,364-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58620569] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 58620569, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:10:02,367-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3c29a9e2 >2018-05-30 05:10:02,367-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:10:02,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:10:02,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:10:02,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3c29a9e2 >2018-05-30 05:10:02,465-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:10:02,477-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [2ddb9b4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:10:02,479-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ddb9b4e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c490823 >2018-05-30 05:10:02,481-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [2ddb9b4e] FINISH, SetVdsStatusVDSCommand, log id: 3c490823 >2018-05-30 05:10:02,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ddb9b4e, Job ID: 36ca2780-1d18-48d8-91af-9e75d091e773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:10:02,491-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 29eddee7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:10:02,493-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2ddb9b4e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 29eddee7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:10:02,504-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5df340e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:10:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5df340e4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7adca4e1] Autorecovering 2 hosts >2018-05-30 05:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7adca4e1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7a040cd6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2aff68bd >2018-05-30 05:15:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [7a040cd6] FINISH, SetVdsStatusVDSCommand, log id: 2aff68bd >2018-05-30 05:15:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [7a040cd6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:15:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [7a040cd6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 77d437bc >2018-05-30 05:15:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] FINISH, SetVdsStatusVDSCommand, log id: 77d437bc >2018-05-30 05:15:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4aeaa8aa >2018-05-30 05:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [5601ae6d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4aeaa8aa >2018-05-30 05:15:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:15:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [5601ae6d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:15:02,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [75743d53] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2d66d61d >2018-05-30 05:15:02,634-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [75743d53] FINISH, GetHardwareInfoVDSCommand, log id: 2d66d61d >2018-05-30 05:15:02,647-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5fb7cd74] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:15:02,655-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [483ed0e0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:15:02,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54d14798] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:15:02,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54d14798] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 721752fd >2018-05-30 05:15:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:15:03,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [54d14798] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 721752fd >2018-05-30 05:15:03,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,052-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,056-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,060-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,061-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [54d14798] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,064-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [54d14798] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:15:03,066-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [54d14798] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 54d14798, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:15:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f5ab061 >2018-05-30 05:15:03,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:15:03,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [79b3ec92] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 71dc0de5 >2018-05-30 05:15:03,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [79b3ec92] FINISH, GetHardwareInfoVDSCommand, log id: 71dc0de5 >2018-05-30 05:15:03,134-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6a223e88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:15:03,139-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1d7ef558] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:15:03,144-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7959623d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:15:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7959623d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63730f89 >2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:15:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:15:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:15:03,178-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f5ab061 >2018-05-30 05:15:03,178-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:15:03,192-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5c58d55c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:15:03,193-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c58d55c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4cb913d5 >2018-05-30 05:15:03,195-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5c58d55c] FINISH, SetVdsStatusVDSCommand, log id: 4cb913d5 >2018-05-30 05:15:03,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5c58d55c, Job ID: 638972ee-8806-458e-a845-62cb55f34233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:15:03,232-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 483ed0e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:15:03,234-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5c58d55c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 483ed0e0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:15:03,245-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [f26fca3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:15:03,246-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [f26fca3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:15:03,529-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:15:03,529-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [7959623d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 63730f89 >2018-05-30 05:15:03,533-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,533-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,537-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,538-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:15:03,543-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [7959623d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:15:03,545-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [7959623d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:15:03,548-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7959623d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7959623d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:15:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6604b128 >2018-05-30 05:15:03,551-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:15:03,648-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:15:03,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:15:03,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:15:03,648-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:15:03,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6604b128 >2018-05-30 05:15:03,649-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-5) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:15:03,661-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [593f548a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:15:03,662-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [593f548a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e9fdc93 >2018-05-30 05:15:03,664-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [593f548a] FINISH, SetVdsStatusVDSCommand, log id: 7e9fdc93 >2018-05-30 05:15:03,670-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 593f548a, Job ID: 0a157851-8341-4a21-b2fb-730e0d297f3a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:15:03,675-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1d7ef558, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:15:03,677-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [593f548a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1d7ef558, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:15:03,687-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [7e2dee54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:15:03,688-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7e2dee54] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5935ee77] Autorecovering 2 hosts >2018-05-30 05:20:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [5935ee77] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:20:00,012-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:20:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:20:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,019-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,021-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f71786] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71086562 >2018-05-30 05:20:00,023-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [f71786] FINISH, SetVdsStatusVDSCommand, log id: 71086562 >2018-05-30 05:20:00,025-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,026-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [f71786] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:20:00,026-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [f71786] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:20:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:20:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:20:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3707aee8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2da8f045 >2018-05-30 05:20:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [3707aee8] FINISH, SetVdsStatusVDSCommand, log id: 2da8f045 >2018-05-30 05:20:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3707aee8] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3c046754 >2018-05-30 05:20:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [3707aee8] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3c046754 >2018-05-30 05:20:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:20:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [3707aee8] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:20:00,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6add5587] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 585b6231 >2018-05-30 05:20:00,890-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6add5587] FINISH, GetHardwareInfoVDSCommand, log id: 585b6231 >2018-05-30 05:20:00,903-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4c39db13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:20:00,907-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [3821f8f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:20:00,912-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [45274358] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:20:00,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [45274358] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 250d8ff1 >2018-05-30 05:20:01,261-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [456037d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 41fea7d5 >2018-05-30 05:20:01,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [456037d] FINISH, GetHardwareInfoVDSCommand, log id: 41fea7d5 >2018-05-30 05:20:01,275-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [7839b6e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:20:01,279-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [495c6d7e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:20:01,285-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1dcda842] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:20:01,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:20:01,290-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [45274358] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 250d8ff1 >2018-05-30 05:20:01,292-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1dcda842] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41356f43 >2018-05-30 05:20:01,294-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,294-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,299-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,303-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [45274358] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,306-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [45274358] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:20:01,309-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [45274358] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 45274358, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:20:01,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 117f218e >2018-05-30 05:20:01,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:20:01,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:20:01,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:20:01,414-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 117f218e >2018-05-30 05:20:01,414-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:20:01,426-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [76f4520c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:20:01,428-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76f4520c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c7addbe >2018-05-30 05:20:01,430-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [76f4520c] FINISH, SetVdsStatusVDSCommand, log id: 7c7addbe >2018-05-30 05:20:01,462-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 76f4520c, Job ID: 8524496c-09ee-44af-ab0f-6596462b9b89, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:20:01,467-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3821f8f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:20:01,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76f4520c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3821f8f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:20:01,480-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [38e395ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:20:01,481-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [38e395ac] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:20:01,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:20:01,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [1dcda842] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41356f43 >2018-05-30 05:20:01,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,681-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,686-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:20:01,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [1dcda842] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:20:01,694-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [1dcda842] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:20:01,697-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1dcda842] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1dcda842, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:20:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4d289cbd >2018-05-30 05:20:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:20:01,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:20:01,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:20:01,797-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4d289cbd >2018-05-30 05:20:01,797-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:20:01,810-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [55f40f20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:20:01,811-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55f40f20] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 93b5b9a >2018-05-30 05:20:01,813-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [55f40f20] FINISH, SetVdsStatusVDSCommand, log id: 93b5b9a >2018-05-30 05:20:01,818-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 55f40f20, Job ID: 1f71179a-1049-4469-b4c8-593d54cd8dd2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:20:01,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 495c6d7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:20:01,825-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [55f40f20] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 495c6d7e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:20:01,836-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [75490a9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:20:01,837-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [75490a9d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [72dad077] Autorecovering 2 hosts >2018-05-30 05:25:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [72dad077] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:25:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [70a2f4e3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72e488c4 >2018-05-30 05:25:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [70a2f4e3] FINISH, SetVdsStatusVDSCommand, log id: 72e488c4 >2018-05-30 05:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [70a2f4e3] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:25:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [70a2f4e3] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:25:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5345fc19] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e02b7f9 >2018-05-30 05:25:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [5345fc19] FINISH, SetVdsStatusVDSCommand, log id: 6e02b7f9 >2018-05-30 05:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5345fc19] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 51107af4 >2018-05-30 05:25:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [5345fc19] FINISH, SetHaMaintenanceModeVDSCommand, log id: 51107af4 >2018-05-30 05:25:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:25:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [5345fc19] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:25:02,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5811269b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1d9a5669 >2018-05-30 05:25:02,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5811269b] FINISH, GetHardwareInfoVDSCommand, log id: 1d9a5669 >2018-05-30 05:25:02,197-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [638b9564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:25:02,205-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [4d3569c1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:25:02,210-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3031b5b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:25:02,264-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3031b5b5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 74efdaf2 >2018-05-30 05:25:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1eee0dbb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 48e22591 >2018-05-30 05:25:02,397-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [1eee0dbb] FINISH, GetHardwareInfoVDSCommand, log id: 48e22591 >2018-05-30 05:25:02,410-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [205c2c71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:25:02,415-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [73622248] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:25:02,420-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29f1293d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:25:02,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29f1293d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4de447bf >2018-05-30 05:25:02,641-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:25:02,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [3031b5b5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 74efdaf2 >2018-05-30 05:25:02,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,646-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,650-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,653-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,654-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [3031b5b5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,657-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [3031b5b5] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:25:02,659-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3031b5b5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3031b5b5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:25:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 68cf6cb1 >2018-05-30 05:25:02,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:25:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:25:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:25:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 68cf6cb1 >2018-05-30 05:25:02,765-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:25:02,778-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [61543945] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:25:02,779-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [61543945] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 23c86e77 >2018-05-30 05:25:02,782-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [61543945] FINISH, SetVdsStatusVDSCommand, log id: 23c86e77 >2018-05-30 05:25:02,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:25:02,800-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [29f1293d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 4de447bf >2018-05-30 05:25:02,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,806-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,810-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:25:02,815-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [29f1293d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:25:02,816-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 61543945, Job ID: 0c5fd6e7-22f5-41ab-b13a-18e1c0bda662, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:25:02,819-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [29f1293d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:25:02,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4d3569c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:25:02,821-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [29f1293d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29f1293d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:25:02,823-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [61543945] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4d3569c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:25:02,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 64e85e38 >2018-05-30 05:25:02,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:25:02,835-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [b46e17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:25:02,836-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [b46e17] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:25:02,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 64e85e38 >2018-05-30 05:25:02,921-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:25:02,935-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [159b380] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:25:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [159b380] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 573a02d1 >2018-05-30 05:25:02,939-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [159b380] FINISH, SetVdsStatusVDSCommand, log id: 573a02d1 >2018-05-30 05:25:02,970-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 159b380, Job ID: 69b069e2-f587-41e4-8e70-08d43f4e5e29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:25:02,975-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 73622248, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:25:02,977-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [159b380] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 73622248, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:25:02,989-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [4f6dc3ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:25:02,990-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [4f6dc3ad] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [50dbb572] Autorecovering 2 hosts >2018-05-30 05:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [50dbb572] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:30:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [359478ac] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 764e2486 >2018-05-30 05:30:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [359478ac] FINISH, SetVdsStatusVDSCommand, log id: 764e2486 >2018-05-30 05:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [359478ac] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:30:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [359478ac] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:30:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:30:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4036db81] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e290e0 >2018-05-30 05:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [4036db81] FINISH, SetVdsStatusVDSCommand, log id: 54e290e0 >2018-05-30 05:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4036db81] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 729bc049 >2018-05-30 05:30:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [4036db81] FINISH, SetHaMaintenanceModeVDSCommand, log id: 729bc049 >2018-05-30 05:30:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:30:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [4036db81] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:30:00,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [30eba73f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4b25a939 >2018-05-30 05:30:00,541-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [30eba73f] FINISH, GetHardwareInfoVDSCommand, log id: 4b25a939 >2018-05-30 05:30:00,569-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [60099d65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:30:00,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [43cc5a80] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 67a27a9c >2018-05-30 05:30:00,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [43cc5a80] FINISH, GetHardwareInfoVDSCommand, log id: 67a27a9c >2018-05-30 05:30:00,578-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [400ccc10] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:30:00,584-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [133b5a7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:30:00,587-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [395ddc4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:30:00,596-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [d239ab8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:30:00,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [133b5a7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3160fe05 >2018-05-30 05:30:00,601-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1890df75] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:30:00,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1890df75] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 36f642be >2018-05-30 05:30:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:30:01,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [133b5a7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3160fe05 >2018-05-30 05:30:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:30:01,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1890df75] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 36f642be >2018-05-30 05:30:01,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,985-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:01,989-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,990-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [133b5a7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:01,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,992-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:01,993-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [133b5a7d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:30:01,995-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [133b5a7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 133b5a7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:30:01,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:30:01,997-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1890df75] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:30:02,000-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1890df75] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:30:02,002-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1890df75] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1890df75, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:30:02,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3808f16a >2018-05-30 05:30:02,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:30:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:30:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:30:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3808f16a >2018-05-30 05:30:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:30:02,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 545293e3 >2018-05-30 05:30:02,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:30:02,113-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [5ff70546] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:30:02,114-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ff70546] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5719fc29 >2018-05-30 05:30:02,116-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ff70546] FINISH, SetVdsStatusVDSCommand, log id: 5719fc29 >2018-05-30 05:30:02,146-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ff70546, Job ID: 7524560f-7f81-457b-b27b-fd432a8dba23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:30:02,151-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 400ccc10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:30:02,153-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5ff70546] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 400ccc10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:30:02,163-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [58bc5a3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:30:02,164-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [58bc5a3d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:30:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:30:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:30:02,205-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, ConnectStoragePoolVDSCommand, log id: 545293e3 >2018-05-30 05:30:02,205-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-3) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:30:02,218-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [5091cc6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:30:02,219-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5091cc6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1ac92743 >2018-05-30 05:30:02,221-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5091cc6c] FINISH, SetVdsStatusVDSCommand, log id: 1ac92743 >2018-05-30 05:30:02,226-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5091cc6c, Job ID: acbef4f0-6d30-430d-bf48-a9965f9f2e00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:30:02,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: d239ab8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:30:02,233-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5091cc6c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: d239ab8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:30:02,243-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [31c81795] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:30:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [31c81795] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:35:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [41241d02] Autorecovering 2 hosts >2018-05-30 05:35:00,005-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [41241d02] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:35:00,034-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,040-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,042-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1951a101] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1a22d284 >2018-05-30 05:35:00,045-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [1951a101] FINISH, SetVdsStatusVDSCommand, log id: 1a22d284 >2018-05-30 05:35:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [1951a101] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:35:00,047-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [1951a101] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:35:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,060-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7e24cbd9 >2018-05-30 05:35:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] FINISH, SetVdsStatusVDSCommand, log id: 7e24cbd9 >2018-05-30 05:35:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: d5db332 >2018-05-30 05:35:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [5ea6738e] FINISH, SetHaMaintenanceModeVDSCommand, log id: d5db332 >2018-05-30 05:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:35:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5ea6738e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:35:02,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [50ea5884] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2cad5603 >2018-05-30 05:35:02,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [50ea5884] FINISH, GetHardwareInfoVDSCommand, log id: 2cad5603 >2018-05-30 05:35:02,780-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [14e1da4c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:35:02,785-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [13a7225e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:35:02,790-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [281ffc46] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:35:02,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [281ffc46] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29b1b75f >2018-05-30 05:35:02,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6323ba6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7fc13d47 >2018-05-30 05:35:02,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [6323ba6] FINISH, GetHardwareInfoVDSCommand, log id: 7fc13d47 >2018-05-30 05:35:02,951-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [3f81543c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:35:02,955-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [3551fb78] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:35:02,960-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bac7355] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:35:03,022-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bac7355] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5361c6a >2018-05-30 05:35:03,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:35:03,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [281ffc46] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 29b1b75f >2018-05-30 05:35:03,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,187-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,191-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,191-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,196-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [281ffc46] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,199-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [281ffc46] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:35:03,202-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [281ffc46] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 281ffc46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:35:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 22a87390 >2018-05-30 05:35:03,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:35:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:35:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:35:03,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 22a87390 >2018-05-30 05:35:03,305-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-33) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:35:03,319-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [5684fc99] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:35:03,320-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5684fc99] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 22d5e850 >2018-05-30 05:35:03,322-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5684fc99] FINISH, SetVdsStatusVDSCommand, log id: 22d5e850 >2018-05-30 05:35:03,330-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5684fc99, Job ID: ec745064-bbc0-4023-88b8-dc31f2943056, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:35:03,334-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 13a7225e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:35:03,336-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5684fc99] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 13a7225e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:35:03,347-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7177edb0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:35:03,348-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7177edb0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:35:03,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:35:03,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [bac7355] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5361c6a >2018-05-30 05:35:03,398-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,399-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,403-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,404-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:35:03,408-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [bac7355] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:35:03,411-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [bac7355] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:35:03,414-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [bac7355] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: bac7355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:35:03,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ddf9e1f >2018-05-30 05:35:03,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:35:03,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:35:03,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:35:03,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-50) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ddf9e1f >2018-05-30 05:35:03,538-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-50) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:35:03,552-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [78d6fd3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:35:03,553-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [78d6fd3f] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 60e97791 >2018-05-30 05:35:03,555-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [78d6fd3f] FINISH, SetVdsStatusVDSCommand, log id: 60e97791 >2018-05-30 05:35:03,561-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 78d6fd3f, Job ID: 802b1af5-903f-476a-8e8e-a8a5d2eb7c35, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:35:03,565-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 3551fb78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:35:03,567-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [78d6fd3f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 3551fb78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:35:03,578-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [21caba26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:35:03,579-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [21caba26] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5232af6b] Autorecovering 2 hosts >2018-05-30 05:40:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5232af6b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:40:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68cde0e1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c2ad78d >2018-05-30 05:40:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68cde0e1] FINISH, SetVdsStatusVDSCommand, log id: 7c2ad78d >2018-05-30 05:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68cde0e1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:40:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [68cde0e1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:40:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [47922f50] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 75010a41 >2018-05-30 05:40:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [47922f50] FINISH, SetVdsStatusVDSCommand, log id: 75010a41 >2018-05-30 05:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [47922f50] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 49af71ce >2018-05-30 05:40:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [47922f50] FINISH, SetHaMaintenanceModeVDSCommand, log id: 49af71ce >2018-05-30 05:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:40:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [47922f50] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:40:01,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72f47282] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 74856397 >2018-05-30 05:40:01,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [72f47282] FINISH, GetHardwareInfoVDSCommand, log id: 74856397 >2018-05-30 05:40:01,029-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [7589a5c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:40:01,038-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [7ccd8632] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:40:01,043-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2fdbde0b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:40:01,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2fdbde0b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 786a5a50 >2018-05-30 05:40:01,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27a83f1d] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 794a6d3b >2018-05-30 05:40:01,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [27a83f1d] FINISH, GetHardwareInfoVDSCommand, log id: 794a6d3b >2018-05-30 05:40:01,258-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [2eae53fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:40:01,262-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [2fe1ddbc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:40:01,267-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [515f68f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:40:01,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [515f68f] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: ecb2c3c >2018-05-30 05:40:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:40:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2fdbde0b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 786a5a50 >2018-05-30 05:40:01,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,437-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,441-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,444-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,445-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [2fdbde0b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,447-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2fdbde0b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:40:01,450-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2fdbde0b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2fdbde0b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:40:01,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 395f80fc >2018-05-30 05:40:01,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:40:01,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:40:01,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:40:01,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-16) [] FINISH, ConnectStoragePoolVDSCommand, log id: 395f80fc >2018-05-30 05:40:01,554-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-16) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:40:01,567-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2efcd448] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:40:01,568-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2efcd448] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4822c41e >2018-05-30 05:40:01,570-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2efcd448] FINISH, SetVdsStatusVDSCommand, log id: 4822c41e >2018-05-30 05:40:01,602-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2efcd448, Job ID: a5e0e29b-ace9-436b-aaf0-12989588961c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:40:01,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7ccd8632, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:40:01,609-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2efcd448] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7ccd8632, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:40:01,619-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [36c5cf5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:40:01,620-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [36c5cf5e] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:40:01,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:40:01,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [515f68f] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: ecb2c3c >2018-05-30 05:40:01,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,651-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,656-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:40:01,660-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [515f68f] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:40:01,663-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [515f68f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:40:01,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [515f68f] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 515f68f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:40:01,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 35dca898 >2018-05-30 05:40:01,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:40:01,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:40:01,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:40:01,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-42) [] FINISH, ConnectStoragePoolVDSCommand, log id: 35dca898 >2018-05-30 05:40:01,765-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-42) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:40:01,778-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [18b18a57] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:40:01,779-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18b18a57] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2eceaa6 >2018-05-30 05:40:01,781-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [18b18a57] FINISH, SetVdsStatusVDSCommand, log id: 2eceaa6 >2018-05-30 05:40:01,786-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 18b18a57, Job ID: 3084ecd0-6c89-4c02-b6c8-e25b19e1a361, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:40:01,791-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2fe1ddbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:40:01,793-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18b18a57] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2fe1ddbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:40:01,803-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [1e081033] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:40:01,804-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1e081033] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6ace51fa] Autorecovering 2 hosts >2018-05-30 05:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6ace51fa] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:45:00,036-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:45:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:45:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,044-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a19688] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 78b9db37 >2018-05-30 05:45:00,047-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2a19688] FINISH, SetVdsStatusVDSCommand, log id: 78b9db37 >2018-05-30 05:45:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2a19688] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:45:00,050-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [2a19688] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:45:00,056-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 556b98f9 >2018-05-30 05:45:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] FINISH, SetVdsStatusVDSCommand, log id: 556b98f9 >2018-05-30 05:45:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 259ea70c >2018-05-30 05:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [2bad4c36] FINISH, SetHaMaintenanceModeVDSCommand, log id: 259ea70c >2018-05-30 05:45:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:45:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [2bad4c36] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:45:02,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [658d8ca8] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6667a756 >2018-05-30 05:45:02,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [658d8ca8] FINISH, GetHardwareInfoVDSCommand, log id: 6667a756 >2018-05-30 05:45:02,215-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [67765d21] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:45:02,223-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5e315b00] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:45:02,228-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [497c1251] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:45:02,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [497c1251] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2ebba05f >2018-05-30 05:45:02,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:45:02,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [497c1251] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2ebba05f >2018-05-30 05:45:02,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:02,629-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:02,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:02,634-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:02,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:02,638-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [497c1251] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:02,641-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [497c1251] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:45:02,644-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [497c1251] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 497c1251, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:45:02,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5935c0e8 >2018-05-30 05:45:02,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:45:02,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [79e08d18] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 687638e7 >2018-05-30 05:45:02,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [79e08d18] FINISH, GetHardwareInfoVDSCommand, log id: 687638e7 >2018-05-30 05:45:02,740-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [3d13a5b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:45:02,744-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [85fc816] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:45:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:45:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:45:02,749-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5935c0e8 >2018-05-30 05:45:02,749-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:45:02,750-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [559dae05] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:45:02,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [559dae05] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 55bba857 >2018-05-30 05:45:02,766-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [2ff8ddeb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:45:02,767-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2ff8ddeb] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b78d972 >2018-05-30 05:45:02,769-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2ff8ddeb] FINISH, SetVdsStatusVDSCommand, log id: 4b78d972 >2018-05-30 05:45:02,777-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ff8ddeb, Job ID: 97fcb827-ec8f-4058-953f-909ea4265774, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:45:02,782-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5e315b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:45:02,784-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2ff8ddeb] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5e315b00, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:45:02,795-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [6880039c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:45:02,796-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6880039c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:45:03,128-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:45:03,129-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [559dae05] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 55bba857 >2018-05-30 05:45:03,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:03,133-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:03,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:03,137-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:03,141-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:45:03,142-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [559dae05] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:45:03,145-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [559dae05] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:45:03,148-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [559dae05] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 559dae05, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:45:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3fa3a718 >2018-05-30 05:45:03,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:45:03,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:45:03,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:45:03,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3fa3a718 >2018-05-30 05:45:03,250-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-18) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:45:03,263-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [7878061c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:45:03,264-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7878061c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 81eaa92 >2018-05-30 05:45:03,267-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [7878061c] FINISH, SetVdsStatusVDSCommand, log id: 81eaa92 >2018-05-30 05:45:03,272-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7878061c, Job ID: d72886de-c8da-49a9-9d2a-a1bed3ada8e4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:45:03,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 85fc816, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:45:03,280-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7878061c] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 85fc816, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:45:03,290-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [2de7d0f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:45:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [2de7d0f] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3fbf2294] Autorecovering 2 hosts >2018-05-30 05:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3fbf2294] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:50:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70a6a9aa] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 28510106 >2018-05-30 05:50:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [70a6a9aa] FINISH, SetVdsStatusVDSCommand, log id: 28510106 >2018-05-30 05:50:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [70a6a9aa] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:50:00,052-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [70a6a9aa] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:50:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:50:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:50:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,111-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,113-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [263e0d76] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 27f0633c >2018-05-30 05:50:00,116-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [263e0d76] FINISH, SetVdsStatusVDSCommand, log id: 27f0633c >2018-05-30 05:50:00,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [263e0d76] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6ff54187 >2018-05-30 05:50:00,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [263e0d76] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6ff54187 >2018-05-30 05:50:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:50:00,122-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [263e0d76] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:50:00,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2baa4e5f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5b381780 >2018-05-30 05:50:00,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2baa4e5f] FINISH, GetHardwareInfoVDSCommand, log id: 5b381780 >2018-05-30 05:50:00,420-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6a8265ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:50:00,428-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [632a55a0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:50:00,433-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a9c30c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:50:00,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a9c30c1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b1df337 >2018-05-30 05:50:00,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:50:00,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [1a9c30c1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b1df337 >2018-05-30 05:50:00,881-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:00,881-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:00,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:00,885-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:00,889-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:00,890-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [1a9c30c1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:00,893-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [1a9c30c1] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:50:00,896-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1a9c30c1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1a9c30c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:50:00,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 792b848c >2018-05-30 05:50:00,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:50:00,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [11c900ad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 69a048fe >2018-05-30 05:50:00,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [11c900ad] FINISH, GetHardwareInfoVDSCommand, log id: 69a048fe >2018-05-30 05:50:00,921-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [f86259b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:50:00,926-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [4bd2fd65] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:50:00,931-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6c6687fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:50:00,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6c6687fb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5365bc45 >2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:50:01,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:50:01,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:50:01,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 792b848c >2018-05-30 05:50:01,003-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:50:01,015-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2e2616bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:50:01,017-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2e2616bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cdce96e >2018-05-30 05:50:01,019-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2e2616bf] FINISH, SetVdsStatusVDSCommand, log id: cdce96e >2018-05-30 05:50:01,027-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e2616bf, Job ID: 4430a226-d629-4502-a591-b4b17da7d57c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:50:01,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 632a55a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:50:01,033-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e2616bf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 632a55a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:50:01,044-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [d4e21b2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:50:01,045-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [d4e21b2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:50:01,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:50:01,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [6c6687fb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5365bc45 >2018-05-30 05:50:01,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:01,314-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:01,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:01,318-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:01,322-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:50:01,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [6c6687fb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:50:01,325-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [6c6687fb] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:50:01,328-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c6687fb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6c6687fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:50:01,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4b9870ca >2018-05-30 05:50:01,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:50:01,432-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:50:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:50:01,432-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:50:01,432-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:50:01,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4b9870ca >2018-05-30 05:50:01,433-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-49) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:50:01,445-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1c5a5592] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:50:01,446-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c5a5592] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5eef7a2a >2018-05-30 05:50:01,448-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1c5a5592] FINISH, SetVdsStatusVDSCommand, log id: 5eef7a2a >2018-05-30 05:50:01,453-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1c5a5592, Job ID: 06004afd-c82a-47c6-9d0a-4df8745caf68, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:50:01,458-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4bd2fd65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:50:01,460-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1c5a5592] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4bd2fd65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:50:01,470-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [79e915f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:50:01,471-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [79e915f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:54:43,557-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [64a61388] Attempting to update VMs/Templates Ovf. >2018-05-30 05:54:43,560-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 05:54:43,560-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 05:54:43,560-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:54:43,565-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 05:54:43,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 05:54:43,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 05:54:43,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 05:54:43,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 05:54:43,569-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 05:54:43,569-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [5daaf15a] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 05:55:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [be79151] Autorecovering 2 hosts >2018-05-30 05:55:00,010-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [be79151] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [154a736c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3672a3c5 >2018-05-30 05:55:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [154a736c] FINISH, SetVdsStatusVDSCommand, log id: 3672a3c5 >2018-05-30 05:55:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [154a736c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 05:55:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [154a736c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:55:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 05:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9769b2b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3b2afeb >2018-05-30 05:55:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [9769b2b] FINISH, SetVdsStatusVDSCommand, log id: 3b2afeb >2018-05-30 05:55:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [9769b2b] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4a94744 >2018-05-30 05:55:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [9769b2b] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4a94744 >2018-05-30 05:55:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 05:55:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [9769b2b] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 05:55:01,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3673234f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 11d13e99 >2018-05-30 05:55:01,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [3673234f] FINISH, GetHardwareInfoVDSCommand, log id: 11d13e99 >2018-05-30 05:55:01,678-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [75e7aff2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:55:01,687-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [7e0a96ed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:55:01,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [d9dbf5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:55:01,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [d9dbf5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f0206b4 >2018-05-30 05:55:02,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:55:02,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [d9dbf5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f0206b4 >2018-05-30 05:55:02,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,081-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,085-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,086-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,090-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [d9dbf5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,093-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [d9dbf5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:55:02,096-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [d9dbf5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: d9dbf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:55:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5d0304db >2018-05-30 05:55:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:55:02,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6d930ea] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6a1e0881 >2018-05-30 05:55:02,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [6d930ea] FINISH, GetHardwareInfoVDSCommand, log id: 6a1e0881 >2018-05-30 05:55:02,153-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7355d217] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:55:02,157-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [340727ea] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:55:02,162-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6386a71e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 05:55:02,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6386a71e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 308848fe >2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:55:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:55:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:55:02,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5d0304db >2018-05-30 05:55:02,197-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:55:02,210-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [16e29c17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:55:02,212-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [16e29c17] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 43c989a >2018-05-30 05:55:02,214-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [16e29c17] FINISH, SetVdsStatusVDSCommand, log id: 43c989a >2018-05-30 05:55:02,245-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 16e29c17, Job ID: 5d6acdb3-e78e-409b-8487-e9b6f7005232, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:55:02,250-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7e0a96ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:55:02,252-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [16e29c17] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7e0a96ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:55:02,263-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [2e87dc70] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 05:55:02,264-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [2e87dc70] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 05:55:02,534-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 05:55:02,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6386a71e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 308848fe >2018-05-30 05:55:02,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,539-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,543-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,547-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 05:55:02,548-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6386a71e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 05:55:02,551-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6386a71e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 05:55:02,554-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6386a71e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6386a71e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 05:55:02,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 546973f >2018-05-30 05:55:02,557-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:55:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 05:55:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 05:55:02,652-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-43) [] FINISH, ConnectStoragePoolVDSCommand, log id: 546973f >2018-05-30 05:55:02,652-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-43) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 05:55:02,666-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [72c6a14e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:55:02,667-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [72c6a14e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: cefb307 >2018-05-30 05:55:02,669-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [72c6a14e] FINISH, SetVdsStatusVDSCommand, log id: cefb307 >2018-05-30 05:55:02,675-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 72c6a14e, Job ID: 23c1e0ae-113a-45cc-bd05-df5f4099d2f2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 05:55:02,679-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 340727ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 05:55:02,682-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [72c6a14e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 340727ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 05:55:02,692-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [31817824] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 05:55:02,693-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [31817824] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7885527] Autorecovering 2 hosts >2018-05-30 06:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7885527] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:00:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7951846c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 45f14d30 >2018-05-30 06:00:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [7951846c] FINISH, SetVdsStatusVDSCommand, log id: 45f14d30 >2018-05-30 06:00:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [7951846c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:00:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7951846c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:00:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4850022e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 310d8d0e >2018-05-30 06:00:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4850022e] FINISH, SetVdsStatusVDSCommand, log id: 310d8d0e >2018-05-30 06:00:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4850022e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 3e356789 >2018-05-30 06:00:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [4850022e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 3e356789 >2018-05-30 06:00:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:00:00,104-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4850022e] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 4850022e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-30 06:00:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [4850022e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:00:00,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [620177ba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 76cc2d6a >2018-05-30 06:00:00,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [620177ba] FINISH, GetHardwareInfoVDSCommand, log id: 76cc2d6a >2018-05-30 06:00:00,323-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [4025e414] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:00:00,328-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [2bab774a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:00:00,333-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [18affc7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:00:00,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [18affc7d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 430919bc >2018-05-30 06:00:00,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:00:00,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [18affc7d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 430919bc >2018-05-30 06:00:00,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:00,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:00,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:00,722-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:00,726-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:00,726-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [18affc7d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:00,729-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [18affc7d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:00:00,732-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [18affc7d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 18affc7d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:00:00,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3907417f >2018-05-30 06:00:00,735-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:00:00,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:00:00,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:00:00,836-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3907417f >2018-05-30 06:00:00,836-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-9) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:00:00,850-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [490886f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:00:00,851-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [490886f0] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ee2866e >2018-05-30 06:00:00,853-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [490886f0] FINISH, SetVdsStatusVDSCommand, log id: ee2866e >2018-05-30 06:00:00,882-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 490886f0, Job ID: 40126c1b-05c7-46e1-bf56-240b6e77d0ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:00:00,887-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2bab774a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:00:00,889-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [490886f0] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2bab774a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:00:00,899-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [1c4b5051] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:00:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [1c4b5051] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:00:02,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [678ca7ba] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1e57a4e >2018-05-30 06:00:02,860-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [678ca7ba] FINISH, GetHardwareInfoVDSCommand, log id: 1e57a4e >2018-05-30 06:00:02,873-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [73e8c13d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:00:02,882-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [313139bb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:00:02,887-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1eb0a492] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:00:02,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1eb0a492] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3c14ef7f >2018-05-30 06:00:03,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:00:03,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [1eb0a492] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3c14ef7f >2018-05-30 06:00:03,273-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:03,274-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:03,277-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:03,278-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:03,282-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:00:03,282-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [1eb0a492] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:00:03,285-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [1eb0a492] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:00:03,288-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1eb0a492] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 1eb0a492, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:00:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 76e77f6b >2018-05-30 06:00:03,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:00:03,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:00:03,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:00:03,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: 76e77f6b >2018-05-30 06:00:03,384-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:00:03,397-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [27350676] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:00:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [27350676] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c15236c >2018-05-30 06:00:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [27350676] FINISH, SetVdsStatusVDSCommand, log id: 2c15236c >2018-05-30 06:00:03,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 27350676, Job ID: 471380ac-9503-4df4-8119-3e7d40233fef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:00:03,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 313139bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:00:03,412-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [27350676] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 313139bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:00:03,423-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [11525346] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:00:03,424-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [11525346] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6566e070] Autorecovering 2 hosts >2018-05-30 06:05:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [6566e070] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:05:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3632109c] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bb6e5b5 >2018-05-30 06:05:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [3632109c] FINISH, SetVdsStatusVDSCommand, log id: 3bb6e5b5 >2018-05-30 06:05:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,079-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3632109c] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 3632109c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com was autorecovered. >2018-05-30 06:05:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [3632109c] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:05:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [3632109c] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:05:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55427bd3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 54e470e3 >2018-05-30 06:05:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [55427bd3] FINISH, SetVdsStatusVDSCommand, log id: 54e470e3 >2018-05-30 06:05:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [55427bd3] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4f927111 >2018-05-30 06:05:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [55427bd3] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4f927111 >2018-05-30 06:05:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:05:00,101-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [55427bd3] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:05:01,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b8336f7] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413ecc88 >2018-05-30 06:05:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1b8336f7] FINISH, GetHardwareInfoVDSCommand, log id: 413ecc88 >2018-05-30 06:05:01,100-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1b786b87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:05:01,109-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [729957f9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:05:01,114-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2a5461db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:05:01,175-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2a5461db] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7031c34a >2018-05-30 06:05:01,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:05:01,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [2a5461db] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7031c34a >2018-05-30 06:05:01,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:01,562-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:01,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:01,567-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:01,570-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:01,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [2a5461db] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:01,574-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [2a5461db] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:05:01,577-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2a5461db] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2a5461db, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:05:01,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3a1640af >2018-05-30 06:05:01,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:05:01,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4964136e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 37393309 >2018-05-30 06:05:01,617-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [4964136e] FINISH, GetHardwareInfoVDSCommand, log id: 37393309 >2018-05-30 06:05:01,630-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7d887db5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:05:01,634-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [53e2783e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:05:01,639-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5a39b582] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:05:01,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5a39b582] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 19abaf41 >2018-05-30 06:05:01,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:05:01,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:05:01,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:05:01,683-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:05:01,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3a1640af >2018-05-30 06:05:01,683-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:05:01,695-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [f0acb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:05:01,696-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [f0acb1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49d92a18 >2018-05-30 06:05:01,698-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [f0acb1] FINISH, SetVdsStatusVDSCommand, log id: 49d92a18 >2018-05-30 06:05:01,729-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: f0acb1, Job ID: f13ca29a-c1a4-4d4d-9ba8-cd195a71798f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:05:01,733-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 729957f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:05:01,735-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [f0acb1] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 729957f9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:05:01,745-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [532ae93d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:05:01,746-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [532ae93d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:05:02,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:05:02,017-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [5a39b582] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 19abaf41 >2018-05-30 06:05:02,021-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:02,021-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:02,025-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:02,026-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:02,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:05:02,031-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [5a39b582] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:05:02,034-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [5a39b582] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:05:02,036-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5a39b582] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5a39b582, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:05:02,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5cef7c3a >2018-05-30 06:05:02,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:05:02,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:05:02,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:05:02,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5cef7c3a >2018-05-30 06:05:02,138-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:05:02,151-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [9130d92] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:05:02,152-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [9130d92] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 165b96d7 >2018-05-30 06:05:02,154-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [9130d92] FINISH, SetVdsStatusVDSCommand, log id: 165b96d7 >2018-05-30 06:05:02,160-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 9130d92, Job ID: d880b4bb-af10-4725-b832-0592dd62b355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:05:02,164-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 53e2783e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:05:02,167-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9130d92] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 53e2783e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:05:02,177-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [253364f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:05:02,179-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [253364f2] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b9dabdc] Autorecovering 2 hosts >2018-05-30 06:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [6b9dabdc] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:10:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [46d8ca6a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71b9d456 >2018-05-30 06:10:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [46d8ca6a] FINISH, SetVdsStatusVDSCommand, log id: 71b9d456 >2018-05-30 06:10:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [46d8ca6a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:10:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [46d8ca6a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:10:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [77d55822] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 8b426bf >2018-05-30 06:10:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [77d55822] FINISH, SetVdsStatusVDSCommand, log id: 8b426bf >2018-05-30 06:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [77d55822] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1643f28 >2018-05-30 06:10:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [77d55822] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1643f28 >2018-05-30 06:10:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:10:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [77d55822] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:10:02,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [253f4490] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2fd5a23c >2018-05-30 06:10:02,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [253f4490] FINISH, GetHardwareInfoVDSCommand, log id: 2fd5a23c >2018-05-30 06:10:02,410-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [1c2bbe77] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:10:02,419-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [36d039cd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:10:02,425-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2bdc56a5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:10:02,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2bdc56a5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 448c6e15 >2018-05-30 06:10:02,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:10:02,819-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [2bdc56a5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 448c6e15 >2018-05-30 06:10:02,821-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ac40a85] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 5015ee6f >2018-05-30 06:10:02,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [5ac40a85] FINISH, GetHardwareInfoVDSCommand, log id: 5015ee6f >2018-05-30 06:10:02,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:02,826-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:02,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:02,830-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:02,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:02,835-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [2bdc56a5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:02,837-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [65e58eb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:10:02,839-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [2bdc56a5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:10:02,842-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [41519d0c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:10:02,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2bdc56a5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2bdc56a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:10:02,847-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [a0a8051] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:10:02,849-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 14737746 >2018-05-30 06:10:02,849-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:10:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [a0a8051] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3113168b >2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:10:02,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:10:02,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:10:02,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-27) [] FINISH, ConnectStoragePoolVDSCommand, log id: 14737746 >2018-05-30 06:10:02,948-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-27) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:10:02,961-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1321b2ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:10:02,962-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1321b2ba] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 426e14cb >2018-05-30 06:10:02,964-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1321b2ba] FINISH, SetVdsStatusVDSCommand, log id: 426e14cb >2018-05-30 06:10:02,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1321b2ba, Job ID: 274cd64b-b64c-4125-b53d-30fd4087a285, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:10:02,974-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 36d039cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:10:02,976-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1321b2ba] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 36d039cd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:10:02,986-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [721a73b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:10:02,988-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [721a73b5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:10:03,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:10:03,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [a0a8051] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3113168b >2018-05-30 06:10:03,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:03,231-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:03,234-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:03,235-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:03,239-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:10:03,240-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [a0a8051] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:10:03,243-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [a0a8051] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:10:03,246-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [a0a8051] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: a0a8051, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:10:03,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3d25e958 >2018-05-30 06:10:03,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:10:03,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:10:03,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:10:03,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3d25e958 >2018-05-30 06:10:03,349-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:10:03,363-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [71bd9338] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:10:03,364-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71bd9338] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 10286dc0 >2018-05-30 06:10:03,367-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [71bd9338] FINISH, SetVdsStatusVDSCommand, log id: 10286dc0 >2018-05-30 06:10:03,372-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71bd9338, Job ID: e0d41082-5cc7-4cbb-9172-bddfccbb9dbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:10:03,377-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 41519d0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:10:03,379-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [71bd9338] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 41519d0c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:10:03,390-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [1aee8e99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:10:03,391-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [1aee8e99] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [28b5d657] Autorecovering 2 hosts >2018-05-30 06:15:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [28b5d657] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:15:00,038-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:15:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:15:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,045-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,046-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [68ad9992] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1e8f1044 >2018-05-30 06:15:00,049-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [68ad9992] FINISH, SetVdsStatusVDSCommand, log id: 1e8f1044 >2018-05-30 06:15:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [68ad9992] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:15:00,052-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [68ad9992] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:15:00,057-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:15:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 632393f7 >2018-05-30 06:15:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] FINISH, SetVdsStatusVDSCommand, log id: 632393f7 >2018-05-30 06:15:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 692a1cae >2018-05-30 06:15:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [6b2ba46d] FINISH, SetHaMaintenanceModeVDSCommand, log id: 692a1cae >2018-05-30 06:15:00,072-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:15:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [6b2ba46d] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:15:00,641-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38344d22] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1321586f >2018-05-30 06:15:00,643-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [38344d22] FINISH, GetHardwareInfoVDSCommand, log id: 1321586f >2018-05-30 06:15:00,655-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [10b3db0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:15:00,664-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [17bb5d12] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:15:00,669-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [29374b39] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:15:00,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [29374b39] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 125f2259 >2018-05-30 06:15:01,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2c7dc724] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2bc3039e >2018-05-30 06:15:01,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [2c7dc724] FINISH, GetHardwareInfoVDSCommand, log id: 2bc3039e >2018-05-30 06:15:01,024-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5007ca70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:15:01,032-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5688878f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:15:01,038-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [160e82c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:15:01,050-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [160e82c1] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d12b098 >2018-05-30 06:15:01,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:15:01,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [29374b39] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 125f2259 >2018-05-30 06:15:01,065-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:01,065-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:01,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:01,070-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:01,074-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:01,074-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [29374b39] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:01,078-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [29374b39] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:15:01,080-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [29374b39] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 29374b39, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:15:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1131f2b3 >2018-05-30 06:15:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:15:01,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:15:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:15:01,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:15:01,187-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:15:01,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-8) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1131f2b3 >2018-05-30 06:15:01,187-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-8) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:15:01,199-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [64c85782] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:15:01,200-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [64c85782] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: d4757e7 >2018-05-30 06:15:01,203-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [64c85782] FINISH, SetVdsStatusVDSCommand, log id: d4757e7 >2018-05-30 06:15:01,211-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 64c85782, Job ID: cbbd9137-a02d-4a1d-b9ad-2a13a663d373, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:15:01,216-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17bb5d12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:15:01,218-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [64c85782] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17bb5d12, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:15:01,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [36753e76] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:15:01,230-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [36753e76] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:15:02,413-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:15:02,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [160e82c1] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7d12b098 >2018-05-30 06:15:02,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:02,417-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:02,421-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:02,422-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:02,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:15:02,426-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [160e82c1] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:15:02,429-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [160e82c1] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:15:02,432-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [160e82c1] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 160e82c1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:15:02,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2b9eaa4a >2018-05-30 06:15:02,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:15:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:15:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:15:02,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2b9eaa4a >2018-05-30 06:15:02,530-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:15:02,543-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [226fd8d3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:15:02,544-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226fd8d3] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4067d2d1 >2018-05-30 06:15:02,546-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [226fd8d3] FINISH, SetVdsStatusVDSCommand, log id: 4067d2d1 >2018-05-30 06:15:02,551-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 226fd8d3, Job ID: 426c8d4b-5224-4001-8a37-f2232d1e9f82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:15:02,556-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5688878f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:15:02,558-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [226fd8d3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5688878f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:15:02,569-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [680d3373] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:15:02,570-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [680d3373] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [396b605] Autorecovering 2 hosts >2018-05-30 06:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [396b605] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:20:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49f7e024] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3985150d >2018-05-30 06:20:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [49f7e024] FINISH, SetVdsStatusVDSCommand, log id: 3985150d >2018-05-30 06:20:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [49f7e024] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:20:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [49f7e024] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:20:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [548986fb] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7d0d1ab0 >2018-05-30 06:20:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [548986fb] FINISH, SetVdsStatusVDSCommand, log id: 7d0d1ab0 >2018-05-30 06:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [548986fb] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2a392008 >2018-05-30 06:20:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [548986fb] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2a392008 >2018-05-30 06:20:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:20:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [548986fb] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:20:00,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e52567f] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 281c9a2b >2018-05-30 06:20:00,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [4e52567f] FINISH, GetHardwareInfoVDSCommand, log id: 281c9a2b >2018-05-30 06:20:00,278-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [662de495] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:20:00,286-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [45eb72b1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:20:00,292-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [63e1722c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:20:00,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [63e1722c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2dfa0b06 >2018-05-30 06:20:00,681-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:20:00,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [63e1722c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2dfa0b06 >2018-05-30 06:20:00,685-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:00,685-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:00,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:00,690-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:00,694-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:00,695-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [63e1722c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:00,697-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [63e1722c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:20:00,699-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [63e1722c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 63e1722c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:20:00,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1377c37d >2018-05-30 06:20:00,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:20:00,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:20:00,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:20:00,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:20:00,804-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:20:00,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1377c37d >2018-05-30 06:20:00,804-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:20:00,817-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [62bbf2ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:20:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62bbf2ef] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: c0d39aa >2018-05-30 06:20:00,820-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [62bbf2ef] FINISH, SetVdsStatusVDSCommand, log id: c0d39aa >2018-05-30 06:20:00,854-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62bbf2ef, Job ID: 830a9115-756b-40c0-8806-9d05bfff0667, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:20:00,859-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 45eb72b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:20:00,861-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [62bbf2ef] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 45eb72b1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:20:00,871-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [2569640e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:20:00,872-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [2569640e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:20:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3bd2883] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 835968e >2018-05-30 06:20:01,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [3bd2883] FINISH, GetHardwareInfoVDSCommand, log id: 835968e >2018-05-30 06:20:01,896-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [91acef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:20:01,904-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [404c93c2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:20:01,910-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6269e8c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:20:01,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6269e8c8] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6e3be37b >2018-05-30 06:20:02,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:20:02,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [6269e8c8] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 6e3be37b >2018-05-30 06:20:02,305-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:02,306-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:02,310-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:02,311-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:02,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:20:02,315-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [6269e8c8] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:20:02,318-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [6269e8c8] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:20:02,321-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6269e8c8] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6269e8c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:20:02,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 21e972a1 >2018-05-30 06:20:02,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:20:02,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:20:02,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:20:02,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 21e972a1 >2018-05-30 06:20:02,425-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:20:02,439-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [2ab933a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:20:02,440-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2ab933a3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51014f7 >2018-05-30 06:20:02,443-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [2ab933a3] FINISH, SetVdsStatusVDSCommand, log id: 51014f7 >2018-05-30 06:20:02,476-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ab933a3, Job ID: cc7615ed-e7a8-4c50-b478-3551c0ab553f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:20:02,481-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 404c93c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:20:02,483-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2ab933a3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 404c93c2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:20:02,494-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [5ef6df05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:20:02,495-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [5ef6df05] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:25:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [593fd8b1] Autorecovering 2 hosts >2018-05-30 06:25:00,007-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [593fd8b1] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:25:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5ed34a17] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e4336e4 >2018-05-30 06:25:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [5ed34a17] FINISH, SetVdsStatusVDSCommand, log id: 5e4336e4 >2018-05-30 06:25:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [5ed34a17] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:25:00,082-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [5ed34a17] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:25:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:25:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:25:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,093-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [74fe8376] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58420254 >2018-05-30 06:25:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [74fe8376] FINISH, SetVdsStatusVDSCommand, log id: 58420254 >2018-05-30 06:25:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [74fe8376] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2dbe2d7e >2018-05-30 06:25:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler1) [74fe8376] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2dbe2d7e >2018-05-30 06:25:00,102-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:25:00,104-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler1) [74fe8376] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:25:01,564-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1f1d0ebe] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2da38441 >2018-05-30 06:25:01,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1f1d0ebe] FINISH, GetHardwareInfoVDSCommand, log id: 2da38441 >2018-05-30 06:25:01,579-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [43776a5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:25:01,588-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [7b162c29] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:25:01,593-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7661f50c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:25:01,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7661f50c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 396fc42b >2018-05-30 06:25:01,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:25:01,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7661f50c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 396fc42b >2018-05-30 06:25:01,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:01,982-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:01,986-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:01,986-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:01,991-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:01,991-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7661f50c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:01,994-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7661f50c] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:25:01,998-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7661f50c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7661f50c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:25:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 54f9b9da >2018-05-30 06:25:02,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:25:02,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 54f9b9da >2018-05-30 06:25:02,100-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:25:02,114-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [17434061] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:25:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17434061] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 18418b0 >2018-05-30 06:25:02,117-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [17434061] FINISH, SetVdsStatusVDSCommand, log id: 18418b0 >2018-05-30 06:25:02,150-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 17434061, Job ID: a8e9920d-1ddb-4313-a9f8-6b019d691175, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:25:02,154-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 7b162c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:25:02,157-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [17434061] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 7b162c29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:25:02,168-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [34a4f041] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:25:02,169-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [34a4f041] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:25:03,912-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [41761b57] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 1f2338bc >2018-05-30 06:25:03,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [41761b57] FINISH, GetHardwareInfoVDSCommand, log id: 1f2338bc >2018-05-30 06:25:03,927-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6723ad5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:25:03,932-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [12eddb9f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:25:03,937-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [36a1777c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:25:03,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [36a1777c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 90dc6d4 >2018-05-30 06:25:04,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:25:04,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [36a1777c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 90dc6d4 >2018-05-30 06:25:04,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:04,324-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:04,328-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:04,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:04,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:25:04,333-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [36a1777c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:25:04,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [36a1777c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:25:04,338-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [36a1777c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 36a1777c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:25:04,342-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 3e32ce28 >2018-05-30 06:25:04,342-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:25:04,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:25:04,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:25:04,439-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-23) [] FINISH, ConnectStoragePoolVDSCommand, log id: 3e32ce28 >2018-05-30 06:25:04,439-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-23) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:25:04,453-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [2190d920] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:25:04,454-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2190d920] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3f69dfe7 >2018-05-30 06:25:04,456-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2190d920] FINISH, SetVdsStatusVDSCommand, log id: 3f69dfe7 >2018-05-30 06:25:04,462-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2190d920, Job ID: fb63bbbb-7ba5-40f1-960a-656ce4b434a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:25:04,466-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 12eddb9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:25:04,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2190d920] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 12eddb9f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:25:04,479-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [4bd21ce4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:25:04,480-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [4bd21ce4] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6daca0f2] Autorecovering 2 hosts >2018-05-30 06:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [6daca0f2] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:30:00,064-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [22512415] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f935a73 >2018-05-30 06:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [22512415] FINISH, SetVdsStatusVDSCommand, log id: 7f935a73 >2018-05-30 06:30:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [22512415] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [22512415] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:30:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c6defec] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 729ac2cd >2018-05-30 06:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2c6defec] FINISH, SetVdsStatusVDSCommand, log id: 729ac2cd >2018-05-30 06:30:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2c6defec] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 6399f10d >2018-05-30 06:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [2c6defec] FINISH, SetHaMaintenanceModeVDSCommand, log id: 6399f10d >2018-05-30 06:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [2c6defec] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:30:02,148-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2d3fe737] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 435fab1f >2018-05-30 06:30:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2d3fe737] FINISH, GetHardwareInfoVDSCommand, log id: 435fab1f >2018-05-30 06:30:02,163-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [5261362c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:30:02,171-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4b8d2cdb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:30:02,176-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7cf4ad9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:30:02,188-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7cf4ad9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 41cb6128 >2018-05-30 06:30:02,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:30:02,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7cf4ad9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 41cb6128 >2018-05-30 06:30:02,571-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:02,571-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:02,576-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:02,576-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:02,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:02,581-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7cf4ad9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:02,584-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7cf4ad9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:30:02,586-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7cf4ad9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7cf4ad9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:30:02,593-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4aff4f01 >2018-05-30 06:30:02,593-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:30:02,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:30:02,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:30:02,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4aff4f01 >2018-05-30 06:30:02,689-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:30:02,702-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [275afff8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:30:02,703-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [275afff8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 349a9988 >2018-05-30 06:30:02,706-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [275afff8] FINISH, SetVdsStatusVDSCommand, log id: 349a9988 >2018-05-30 06:30:02,741-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 275afff8, Job ID: 49be110f-a34a-4ea9-9f99-df4e8408a446, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:30:02,746-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4b8d2cdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:30:02,748-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [275afff8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4b8d2cdb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:30:02,759-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [4ed388b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:30:02,760-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [4ed388b9] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:30:02,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [62818476] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 64430148 >2018-05-30 06:30:02,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [62818476] FINISH, GetHardwareInfoVDSCommand, log id: 64430148 >2018-05-30 06:30:02,882-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [1a922ce3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:30:02,890-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [83d24e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:30:02,895-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [66e8dbfe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:30:02,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [66e8dbfe] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3ee6f520 >2018-05-30 06:30:03,276-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:30:03,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [66e8dbfe] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3ee6f520 >2018-05-30 06:30:03,280-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:03,280-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:03,284-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:03,285-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:03,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:30:03,289-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [66e8dbfe] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:30:03,293-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [66e8dbfe] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:30:03,295-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [66e8dbfe] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 66e8dbfe, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:30:03,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2033f1c9 >2018-05-30 06:30:03,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:30:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:30:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:30:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2033f1c9 >2018-05-30 06:30:03,400-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:30:03,413-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [31ef0b58] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:30:03,415-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [31ef0b58] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 49743a41 >2018-05-30 06:30:03,417-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [31ef0b58] FINISH, SetVdsStatusVDSCommand, log id: 49743a41 >2018-05-30 06:30:03,445-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 31ef0b58, Job ID: eceaa3bd-adaa-47a9-814e-f7bc1a825777, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:30:03,450-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 83d24e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:30:03,452-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31ef0b58] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 83d24e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:30:03,463-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [146f121e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:30:03,464-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [146f121e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2a62c325] Autorecovering 2 hosts >2018-05-30 06:35:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [2a62c325] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:35:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:35:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:35:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,067-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b28141] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4892c060 >2018-05-30 06:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [34b28141] FINISH, SetVdsStatusVDSCommand, log id: 4892c060 >2018-05-30 06:35:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [34b28141] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:35:00,073-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [34b28141] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:35:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f97ce9c >2018-05-30 06:35:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] FINISH, SetVdsStatusVDSCommand, log id: 4f97ce9c >2018-05-30 06:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 5e152abf >2018-05-30 06:35:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [5c9b959e] FINISH, SetHaMaintenanceModeVDSCommand, log id: 5e152abf >2018-05-30 06:35:00,092-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:35:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5c9b959e] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:35:00,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3068b85e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 2df25dc5 >2018-05-30 06:35:00,464-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [3068b85e] FINISH, GetHardwareInfoVDSCommand, log id: 2df25dc5 >2018-05-30 06:35:00,477-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [5c6b7e4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:35:00,485-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [6726d23] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:35:00,490-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7ce8a536] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:35:00,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7ce8a536] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5432c6c6 >2018-05-30 06:35:00,880-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:35:00,880-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [7ce8a536] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5432c6c6 >2018-05-30 06:35:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:00,885-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:00,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:00,889-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:00,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:00,893-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [7ce8a536] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:00,896-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [7ce8a536] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:35:00,898-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [7ce8a536] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7ce8a536, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:35:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2ca22ca4 >2018-05-30 06:35:00,901-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:35:00,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:35:00,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:35:00,998-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2ca22ca4 >2018-05-30 06:35:00,998-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:35:01,011-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [592a3921] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:35:01,012-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [592a3921] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7f0a6546 >2018-05-30 06:35:01,014-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [592a3921] FINISH, SetVdsStatusVDSCommand, log id: 7f0a6546 >2018-05-30 06:35:01,044-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 592a3921, Job ID: ff6b97ac-473a-4861-a3e7-1bf8602fa8f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:35:01,049-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6726d23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:35:01,051-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [592a3921] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6726d23, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:35:01,062-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [7177f2f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:35:01,063-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [7177f2f0] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:35:01,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [79aec68b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 22abe3fc >2018-05-30 06:35:01,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [79aec68b] FINISH, GetHardwareInfoVDSCommand, log id: 22abe3fc >2018-05-30 06:35:01,196-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [6e7061bc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:35:01,201-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [c82c31f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:35:01,205-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [41ec06b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:35:01,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [41ec06b0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78fd3fc8 >2018-05-30 06:35:03,532-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:35:03,532-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [41ec06b0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 78fd3fc8 >2018-05-30 06:35:03,536-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:03,537-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:03,541-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:03,541-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:03,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:35:03,546-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [41ec06b0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:35:03,549-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [41ec06b0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:35:03,552-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [41ec06b0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 41ec06b0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:35:03,555-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7940caf9 >2018-05-30 06:35:03,555-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:35:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:35:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:35:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-45) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7940caf9 >2018-05-30 06:35:03,651-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-45) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:35:03,664-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [65a9f388] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:35:03,666-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a9f388] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ad004db >2018-05-30 06:35:03,668-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [65a9f388] FINISH, SetVdsStatusVDSCommand, log id: 5ad004db >2018-05-30 06:35:03,700-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 65a9f388, Job ID: 04444bf0-ea35-44bb-ac59-5df6e0e1a628, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:35:03,705-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: c82c31f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:35:03,707-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [65a9f388] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: c82c31f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:35:03,717-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [218cb8a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:35:03,718-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [218cb8a0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [77aa42a4] Autorecovering 2 hosts >2018-05-30 06:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [77aa42a4] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f16f4a1] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 79ec384e >2018-05-30 06:40:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [6f16f4a1] FINISH, SetVdsStatusVDSCommand, log id: 79ec384e >2018-05-30 06:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [6f16f4a1] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [6f16f4a1] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [237c18b1] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 40d2910d >2018-05-30 06:40:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [237c18b1] FINISH, SetVdsStatusVDSCommand, log id: 40d2910d >2018-05-30 06:40:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [237c18b1] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1d1813b2 >2018-05-30 06:40:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [237c18b1] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1d1813b2 >2018-05-30 06:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:40:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [237c18b1] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:40:01,518-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27fb8b71] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 546a4aa1 >2018-05-30 06:40:01,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [27fb8b71] FINISH, GetHardwareInfoVDSCommand, log id: 546a4aa1 >2018-05-30 06:40:01,532-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [7d716505] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:40:01,537-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [72e80d3e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:40:01,542-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4c9b5ae0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:40:01,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4c9b5ae0] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 70aae753 >2018-05-30 06:40:01,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1ca5071c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 6650d88b >2018-05-30 06:40:01,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1ca5071c] FINISH, GetHardwareInfoVDSCommand, log id: 6650d88b >2018-05-30 06:40:01,737-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [705a5750] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:40:01,741-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [5c4562f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:40:01,746-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7f51e594] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:40:01,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7f51e594] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f11f2eb >2018-05-30 06:40:01,974-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:40:01,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [4c9b5ae0] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 70aae753 >2018-05-30 06:40:01,978-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:01,979-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:01,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:01,983-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:01,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:01,988-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [4c9b5ae0] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:01,991-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [4c9b5ae0] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:40:01,994-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c9b5ae0] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4c9b5ae0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:40:02,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 265de894 >2018-05-30 06:40:02,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:40:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:40:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:40:02,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 265de894 >2018-05-30 06:40:02,099-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:40:02,112-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [62e472a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:40:02,113-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [62e472a8] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53ab147 >2018-05-30 06:40:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [62e472a8] FINISH, SetVdsStatusVDSCommand, log id: 53ab147 >2018-05-30 06:40:02,124-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 62e472a8, Job ID: e93c3d6b-441e-4f59-ba12-084de009ec2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:40:02,125-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:40:02,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7f51e594] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3f11f2eb >2018-05-30 06:40:02,129-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 72e80d3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:40:02,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:02,130-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:02,131-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [62e472a8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 72e80d3e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:40:02,135-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:02,135-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:02,139-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:40:02,140-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7f51e594] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:40:02,143-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [137131c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:40:02,144-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [137131c3] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:40:02,144-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7f51e594] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:40:02,147-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7f51e594] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7f51e594, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:40:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73d6ce3c >2018-05-30 06:40:02,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:40:02,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:40:02,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:40:02,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73d6ce3c >2018-05-30 06:40:02,246-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-2) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:40:02,260-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [580961be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:40:02,262-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [580961be] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2db780c3 >2018-05-30 06:40:02,264-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [580961be] FINISH, SetVdsStatusVDSCommand, log id: 2db780c3 >2018-05-30 06:40:02,273-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 580961be, Job ID: 384ca8f7-6b62-4601-9698-0eed676e01eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:40:02,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c4562f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:40:02,279-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [580961be] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c4562f3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:40:02,290-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [39917e5b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:40:02,291-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [39917e5b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5cbc1b40] Autorecovering 2 hosts >2018-05-30 06:45:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5cbc1b40] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:45:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3bfb13ea] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3bdb62e0 >2018-05-30 06:45:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [3bfb13ea] FINISH, SetVdsStatusVDSCommand, log id: 3bdb62e0 >2018-05-30 06:45:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [3bfb13ea] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:45:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [3bfb13ea] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:45:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [69d49c82] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f492315 >2018-05-30 06:45:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [69d49c82] FINISH, SetVdsStatusVDSCommand, log id: 5f492315 >2018-05-30 06:45:00,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [69d49c82] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 28e09b1b >2018-05-30 06:45:00,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [69d49c82] FINISH, SetHaMaintenanceModeVDSCommand, log id: 28e09b1b >2018-05-30 06:45:00,120-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:45:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [69d49c82] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:45:02,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5210d619] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: db3b505 >2018-05-30 06:45:02,866-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [5210d619] FINISH, GetHardwareInfoVDSCommand, log id: db3b505 >2018-05-30 06:45:02,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bb2cf88] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 700b5aee >2018-05-30 06:45:02,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [5bb2cf88] FINISH, GetHardwareInfoVDSCommand, log id: 700b5aee >2018-05-30 06:45:02,880-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [5e2d4139] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:45:02,889-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [5aac4b94] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:45:02,889-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [ba1a2e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:45:02,894-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [75e6e032] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:45:02,898-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [52060b2b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:45:02,903-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69791422] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:45:02,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [75e6e032] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 500fbc86 >2018-05-30 06:45:02,915-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69791422] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b91d745 >2018-05-30 06:45:03,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:45:03,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [75e6e032] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 500fbc86 >2018-05-30 06:45:03,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:03,292-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:03,295-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:03,296-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:03,300-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:03,301-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [75e6e032] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:03,304-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [75e6e032] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:45:03,306-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [75e6e032] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 75e6e032, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:45:03,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 12ddb0d8 >2018-05-30 06:45:03,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:45:03,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:45:03,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:45:03,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 12ddb0d8 >2018-05-30 06:45:03,407-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:45:03,420-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [48ba65bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:45:03,421-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [48ba65bf] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3388052c >2018-05-30 06:45:03,424-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [48ba65bf] FINISH, SetVdsStatusVDSCommand, log id: 3388052c >2018-05-30 06:45:03,456-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 48ba65bf, Job ID: 583e8170-6bd1-4406-b851-d5f780687ce7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:45:03,461-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5aac4b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:45:03,464-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [48ba65bf] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5aac4b94, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:45:03,474-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [54aa251f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:45:03,475-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [54aa251f] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:45:04,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:45:04,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [69791422] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1b91d745 >2018-05-30 06:45:04,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:04,323-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:04,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:04,328-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:04,331-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:45:04,332-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [69791422] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:45:04,335-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [69791422] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:45:04,337-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [69791422] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 69791422, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:45:04,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 57b82508 >2018-05-30 06:45:04,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:45:04,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:45:04,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:45:04,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57b82508 >2018-05-30 06:45:04,436-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:45:04,448-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [7d6c8e48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:45:04,449-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7d6c8e48] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2a977925 >2018-05-30 06:45:04,451-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [7d6c8e48] FINISH, SetVdsStatusVDSCommand, log id: 2a977925 >2018-05-30 06:45:04,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7d6c8e48, Job ID: da0c7860-5de0-41ad-a2b8-6a2e2bd67e10, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:45:04,461-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 52060b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:45:04,463-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7d6c8e48] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 52060b2b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:45:04,473-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [342394db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:45:04,474-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [342394db] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5f3029bf] Autorecovering 2 hosts >2018-05-30 06:50:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [5f3029bf] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:50:00,060-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:50:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45415bae] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31d29741 >2018-05-30 06:50:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [45415bae] FINISH, SetVdsStatusVDSCommand, log id: 31d29741 >2018-05-30 06:50:00,073-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [45415bae] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:50:00,074-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler9) [45415bae] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:50:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:50:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:50:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,086-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4947fb95] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 508a50b6 >2018-05-30 06:50:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [4947fb95] FINISH, SetVdsStatusVDSCommand, log id: 508a50b6 >2018-05-30 06:50:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4947fb95] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 36830921 >2018-05-30 06:50:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler9) [4947fb95] FINISH, SetHaMaintenanceModeVDSCommand, log id: 36830921 >2018-05-30 06:50:00,094-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:50:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler9) [4947fb95] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:50:01,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3943ad76] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 619942e0 >2018-05-30 06:50:01,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3943ad76] FINISH, GetHardwareInfoVDSCommand, log id: 619942e0 >2018-05-30 06:50:01,079-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [193b281c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:50:01,083-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [62c9272] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:50:01,089-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3b6cbe0d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:50:01,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3b6cbe0d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5b0d23ad >2018-05-30 06:50:01,470-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:50:01,470-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [3b6cbe0d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5b0d23ad >2018-05-30 06:50:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:01,474-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:01,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:01,479-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:01,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:01,483-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [3b6cbe0d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:01,487-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [3b6cbe0d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:50:01,490-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3b6cbe0d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3b6cbe0d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:50:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 342aba40 >2018-05-30 06:50:01,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-17) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:50:01,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:50:01,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:50:01,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-17) [] FINISH, ConnectStoragePoolVDSCommand, log id: 342aba40 >2018-05-30 06:50:01,592-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-17) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:50:01,604-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [5fde0f2e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:50:01,606-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5fde0f2e] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 31bf53ad >2018-05-30 06:50:01,608-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [5fde0f2e] FINISH, SetVdsStatusVDSCommand, log id: 31bf53ad >2018-05-30 06:50:01,614-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5fde0f2e, Job ID: d633be31-02c3-4a4a-9f60-d151c4b1b7c5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:50:01,618-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 62c9272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:50:01,620-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5fde0f2e] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 62c9272, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:50:01,631-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [476e2797] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:50:01,632-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [476e2797] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:50:02,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5efe00e3] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 7e9d54da >2018-05-30 06:50:02,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [5efe00e3] FINISH, GetHardwareInfoVDSCommand, log id: 7e9d54da >2018-05-30 06:50:02,097-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [34231990] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:50:02,101-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [2d537acc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:50:02,107-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2558c7e7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:50:02,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2558c7e7] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 26462a90 >2018-05-30 06:50:02,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:50:02,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2558c7e7] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 26462a90 >2018-05-30 06:50:02,487-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:02,488-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:02,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:02,492-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:02,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:50:02,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2558c7e7] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:50:02,499-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2558c7e7] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:50:02,502-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2558c7e7] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2558c7e7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:50:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 45ef468e >2018-05-30 06:50:02,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:50:02,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:50:02,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:50:02,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 45ef468e >2018-05-30 06:50:02,604-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:50:02,617-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [71cdc935] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:50:02,618-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [71cdc935] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3967b4eb >2018-05-30 06:50:02,620-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [71cdc935] FINISH, SetVdsStatusVDSCommand, log id: 3967b4eb >2018-05-30 06:50:02,626-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 71cdc935, Job ID: ba439923-8c90-4f2a-a06c-b10f1f1fc6cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:50:02,630-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2d537acc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:50:02,632-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [71cdc935] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2d537acc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:50:02,643-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [4257660a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:50:02,644-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [4257660a] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:54:43,570-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [14aa7392] Attempting to update VMs/Templates Ovf. >2018-05-30 06:54:43,572-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 06:54:43,572-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 06:54:43,573-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:54:43,577-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to update VM OVFs in Data Center 'Default' >2018-05-30 06:54:43,580-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully updated VM OVFs in Data Center 'Default' >2018-05-30 06:54:43,580-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to update template OVFs in Data Center 'Default' >2018-05-30 06:54:43,580-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully updated templates OVFs in Data Center 'Default' >2018-05-30 06:54:43,580-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 06:54:43,581-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Successfully removed unneeded template/vm OVFs in Data Center 'Default' >2018-05-30 06:54:43,581-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [6402a08b] Lock freed to object 'EngineLock:{exclusiveLocks='[5b05a987-0149-0267-0212-0000000001df=OVF_UPDATE]', sharedLocks=''}' >2018-05-30 06:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7e83ac54] Autorecovering 2 hosts >2018-05-30 06:55:00,004-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [7e83ac54] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:55:00,030-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:55:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:55:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,037-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3425631b] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7819d24f >2018-05-30 06:55:00,041-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [3425631b] FINISH, SetVdsStatusVDSCommand, log id: 7819d24f >2018-05-30 06:55:00,042-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,043-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [3425631b] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 06:55:00,043-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [3425631b] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:55:00,049-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:55:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 06:55:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,055-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c2da757] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 51c4e11f >2018-05-30 06:55:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5c2da757] FINISH, SetVdsStatusVDSCommand, log id: 51c4e11f >2018-05-30 06:55:00,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [5c2da757] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 76a02f75 >2018-05-30 06:55:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [5c2da757] FINISH, SetHaMaintenanceModeVDSCommand, log id: 76a02f75 >2018-05-30 06:55:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 06:55:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [5c2da757] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 06:55:00,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6b835725] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 29d08b2 >2018-05-30 06:55:00,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6b835725] FINISH, GetHardwareInfoVDSCommand, log id: 29d08b2 >2018-05-30 06:55:00,267-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [4234c60d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:55:00,271-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6a0f8f11] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:55:00,277-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7c50c59d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:55:00,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7c50c59d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 10970781 >2018-05-30 06:55:00,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:55:00,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7c50c59d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 10970781 >2018-05-30 06:55:00,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:00,667-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:00,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:00,671-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:00,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:00,676-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7c50c59d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:00,679-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7c50c59d] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:55:00,681-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7c50c59d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7c50c59d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:55:00,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: c3b8d6 >2018-05-30 06:55:00,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:55:00,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:55:00,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:55:00,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-14) [] FINISH, ConnectStoragePoolVDSCommand, log id: c3b8d6 >2018-05-30 06:55:00,787-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-14) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:55:00,800-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [7cf5adcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:55:00,802-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7cf5adcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: b0a34f >2018-05-30 06:55:00,804-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [7cf5adcc] FINISH, SetVdsStatusVDSCommand, log id: b0a34f >2018-05-30 06:55:00,836-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 7cf5adcc, Job ID: 0da1590c-8147-46e8-8485-db587d6eb328, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:55:00,841-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6a0f8f11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:55:00,843-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7cf5adcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6a0f8f11, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:55:00,854-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [38988f69] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 06:55:00,855-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [38988f69] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 06:55:02,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [602cac2e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 58b98ac >2018-05-30 06:55:02,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [602cac2e] FINISH, GetHardwareInfoVDSCommand, log id: 58b98ac >2018-05-30 06:55:02,229-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [18fc1076] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:55:02,234-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [49653a5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:55:02,239-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7fe2fb4a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 06:55:02,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7fe2fb4a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 24fcfcfd >2018-05-30 06:55:02,686-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 06:55:02,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [7fe2fb4a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 24fcfcfd >2018-05-30 06:55:02,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:02,691-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:02,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:02,696-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:02,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 06:55:02,700-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [7fe2fb4a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 06:55:02,703-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [7fe2fb4a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 06:55:02,705-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7fe2fb4a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7fe2fb4a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 06:55:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 5452bfa8 >2018-05-30 06:55:02,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 06:55:02,810-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:55:02,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 06:55:02,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 06:55:02,810-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 06:55:02,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5452bfa8 >2018-05-30 06:55:02,811-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 06:55:02,824-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [1f31e14a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:55:02,825-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1f31e14a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3357907f >2018-05-30 06:55:02,827-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [1f31e14a] FINISH, SetVdsStatusVDSCommand, log id: 3357907f >2018-05-30 06:55:02,836-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1f31e14a, Job ID: 3c374413-1cb8-402f-b0aa-620e69580298, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 06:55:02,840-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 49653a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 06:55:02,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1f31e14a] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 49653a5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 06:55:02,853-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [9cb3562] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 06:55:02,854-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [9cb3562] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6edc22b5] Autorecovering 2 hosts >2018-05-30 07:00:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [6edc22b5] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:00:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,054-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [67198037] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 73d5e6d >2018-05-30 07:00:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [67198037] FINISH, SetVdsStatusVDSCommand, log id: 73d5e6d >2018-05-30 07:00:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,059-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [67198037] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:00:00,059-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler4) [67198037] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:00:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [78f48851] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 503376ce >2018-05-30 07:00:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [78f48851] FINISH, SetVdsStatusVDSCommand, log id: 503376ce >2018-05-30 07:00:00,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [78f48851] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61911e0a >2018-05-30 07:00:00,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler4) [78f48851] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61911e0a >2018-05-30 07:00:00,080-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:00:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler4) [78f48851] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:00:00,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2ed85cad] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 413de36a >2018-05-30 07:00:00,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [2ed85cad] FINISH, GetHardwareInfoVDSCommand, log id: 413de36a >2018-05-30 07:00:00,514-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [593f3f9f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:00:00,522-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [34c78a72] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:00:00,528-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5d8ae3bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:00:00,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5d8ae3bb] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5773a4b5 >2018-05-30 07:00:00,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:00:00,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [5d8ae3bb] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 5773a4b5 >2018-05-30 07:00:00,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:00,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:00,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:00,916-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:00,920-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:00,921-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [5d8ae3bb] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:00,924-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [5d8ae3bb] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:00:00,927-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5d8ae3bb] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5d8ae3bb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:00:00,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 6ab0468a >2018-05-30 07:00:00,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:00:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:00:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:00:01,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6ab0468a >2018-05-30 07:00:01,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:00:01,037-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [2e062175] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:00:01,038-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e062175] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6b4e83c >2018-05-30 07:00:01,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [2e062175] FINISH, SetVdsStatusVDSCommand, log id: 6b4e83c >2018-05-30 07:00:01,069-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2e062175, Job ID: 0ed01f50-8e45-4436-9625-ae88080d2e82, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:00:01,073-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 34c78a72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:00:01,075-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2e062175] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 34c78a72, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:00:01,086-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [33d3cb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:00:01,087-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [33d3cb2] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:00:01,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [39edb12c] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 358db0b6 >2018-05-30 07:00:01,574-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [39edb12c] FINISH, GetHardwareInfoVDSCommand, log id: 358db0b6 >2018-05-30 07:00:01,586-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [425e43a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:00:01,591-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [5c9881ef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:00:01,596-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2c0182ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:00:01,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2c0182ef] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 592c108f >2018-05-30 07:00:01,971-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:00:01,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [2c0182ef] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 592c108f >2018-05-30 07:00:01,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:01,975-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:01,979-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:01,980-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:01,984-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:00:01,984-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler6) [2c0182ef] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:00:01,988-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [2c0182ef] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:00:01,990-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2c0182ef] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2c0182ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:00:01,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 66796e2d >2018-05-30 07:00:01,993-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:00:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:00:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:00:02,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 66796e2d >2018-05-30 07:00:02,089-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:00:02,101-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler6) [63535dad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:00:02,102-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [63535dad] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33c8c748 >2018-05-30 07:00:02,104-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [63535dad] FINISH, SetVdsStatusVDSCommand, log id: 33c8c748 >2018-05-30 07:00:02,110-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 63535dad, Job ID: 815328b8-a282-42c1-944b-26012c09ee90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:00:02,114-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5c9881ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:00:02,116-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [63535dad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5c9881ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:00:02,126-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [b6f517b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:00:02,127-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [b6f517b] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:05:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7076586f] Autorecovering 2 hosts >2018-05-30 07:05:00,006-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [7076586f] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:05:00,032-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,039-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2d0fb28a] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3c62b900 >2018-05-30 07:05:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [2d0fb28a] FINISH, SetVdsStatusVDSCommand, log id: 3c62b900 >2018-05-30 07:05:00,044-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,046-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [2d0fb28a] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:05:00,046-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [2d0fb28a] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:05:00,052-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,058-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,059-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4472318a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21fbed0f >2018-05-30 07:05:00,062-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [4472318a] FINISH, SetVdsStatusVDSCommand, log id: 21fbed0f >2018-05-30 07:05:00,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4472318a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2ac15420 >2018-05-30 07:05:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler7) [4472318a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2ac15420 >2018-05-30 07:05:00,066-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:05:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler7) [4472318a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:05:01,679-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [53a249ac] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4ecc368a >2018-05-30 07:05:01,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [53a249ac] FINISH, GetHardwareInfoVDSCommand, log id: 4ecc368a >2018-05-30 07:05:01,695-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3081f566] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:05:01,704-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [160d900e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:05:01,709-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21193063] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:05:01,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21193063] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 42c3b44f >2018-05-30 07:05:02,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:05:02,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [21193063] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 42c3b44f >2018-05-30 07:05:02,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:02,111-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:02,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:02,116-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:02,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:02,120-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [21193063] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:02,124-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [21193063] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:05:02,127-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [21193063] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 21193063, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:05:02,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 23bb91b8 >2018-05-30 07:05:02,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:05:02,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:05:02,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:05:02,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:05:02,228-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:05:02,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 23bb91b8 >2018-05-30 07:05:02,229-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:05:02,243-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [560ce9f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:05:02,244-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [560ce9f3] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7c13f5cb >2018-05-30 07:05:02,247-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [560ce9f3] FINISH, SetVdsStatusVDSCommand, log id: 7c13f5cb >2018-05-30 07:05:02,252-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 560ce9f3, Job ID: 3653c4ba-e4d5-4c41-a789-7457289e7009, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:05:02,257-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 160d900e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:05:02,259-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [560ce9f3] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 160d900e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:05:02,270-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [49a8adf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:05:02,271-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [49a8adf] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:05:02,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e603e94] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 6e9177ea >2018-05-30 07:05:02,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [5e603e94] FINISH, GetHardwareInfoVDSCommand, log id: 6e9177ea >2018-05-30 07:05:02,823-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [2499f555] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:05:02,827-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [1159aee3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:05:02,832-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3c97ee73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:05:02,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3c97ee73] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3b9d0942 >2018-05-30 07:05:03,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:05:03,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [3c97ee73] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 3b9d0942 >2018-05-30 07:05:03,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:03,219-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:03,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:03,223-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:03,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:05:03,228-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [3c97ee73] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:05:03,230-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [3c97ee73] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:05:03,233-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c97ee73] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3c97ee73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:05:03,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 171c5e4c >2018-05-30 07:05:03,236-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:05:03,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:05:03,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:05:03,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 171c5e4c >2018-05-30 07:05:03,338-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:05:03,351-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [1101fcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:05:03,352-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1101fcc] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 66e8d1f6 >2018-05-30 07:05:03,354-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [1101fcc] FINISH, SetVdsStatusVDSCommand, log id: 66e8d1f6 >2018-05-30 07:05:03,359-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1101fcc, Job ID: 85cd10e7-c40e-43b9-b4dc-55bf9e49cfc3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:05:03,364-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1159aee3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:05:03,366-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1101fcc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1159aee3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:05:03,376-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [61ea8921] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:05:03,377-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [61ea8921] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [5fa7e7e9] Autorecovering 2 hosts >2018-05-30 07:10:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [5fa7e7e9] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:10:00,065-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [19fc9253] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57c9f1e >2018-05-30 07:10:00,076-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [19fc9253] FINISH, SetVdsStatusVDSCommand, log id: 57c9f1e >2018-05-30 07:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,079-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [19fc9253] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:10:00,079-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [19fc9253] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:10:00,085-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5ef8ec15 >2018-05-30 07:10:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] FINISH, SetVdsStatusVDSCommand, log id: 5ef8ec15 >2018-05-30 07:10:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 1784a3fe >2018-05-30 07:10:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler6) [6d86f3c9] FINISH, SetHaMaintenanceModeVDSCommand, log id: 1784a3fe >2018-05-30 07:10:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:10:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler6) [6d86f3c9] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:10:01,129-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e5f75bd] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 14ba53b5 >2018-05-30 07:10:01,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3e5f75bd] FINISH, GetHardwareInfoVDSCommand, log id: 14ba53b5 >2018-05-30 07:10:01,144-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [53eba1dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:10:01,153-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [6451c212] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:10:01,158-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [537c4aa9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:10:01,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [537c4aa9] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7ba37cfe >2018-05-30 07:10:01,587-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:10:01,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [537c4aa9] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 7ba37cfe >2018-05-30 07:10:01,591-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:01,592-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:01,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:01,596-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:01,600-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:01,601-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [537c4aa9] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:01,603-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [537c4aa9] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:10:01,606-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [537c4aa9] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 537c4aa9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:10:01,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4088a1cb >2018-05-30 07:10:01,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:10:01,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:10:01,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:10:01,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4088a1cb >2018-05-30 07:10:01,710-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-47) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:10:01,738-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [628ef096] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:10:01,739-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [628ef096] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 21273010 >2018-05-30 07:10:01,741-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [628ef096] FINISH, SetVdsStatusVDSCommand, log id: 21273010 >2018-05-30 07:10:01,773-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 628ef096, Job ID: 13bcc270-0a11-4aac-b2ba-9d12d00b3e46, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:10:01,778-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 6451c212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:10:01,780-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [628ef096] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 6451c212, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:10:01,791-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [4b9a1af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:10:01,792-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [4b9a1af] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:10:02,895-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [46de57d5] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 4e8d85ec >2018-05-30 07:10:02,897-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [46de57d5] FINISH, GetHardwareInfoVDSCommand, log id: 4e8d85ec >2018-05-30 07:10:02,911-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [5a7e996c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:10:02,919-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [1a0000b7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:10:02,925-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e2ad266] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:10:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e2ad266] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1c2fc4e2 >2018-05-30 07:10:03,313-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:10:03,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [3e2ad266] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1c2fc4e2 >2018-05-30 07:10:03,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:03,317-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:03,322-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:03,322-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:03,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:10:03,327-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [3e2ad266] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:10:03,329-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [3e2ad266] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:10:03,332-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3e2ad266] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 3e2ad266, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:10:03,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7d26a7d2 >2018-05-30 07:10:03,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:10:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:10:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:10:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7d26a7d2 >2018-05-30 07:10:03,429-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:10:03,442-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [2ced2cbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:10:03,443-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ced2cbc] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 147fe90a >2018-05-30 07:10:03,445-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [2ced2cbc] FINISH, SetVdsStatusVDSCommand, log id: 147fe90a >2018-05-30 07:10:03,451-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2ced2cbc, Job ID: f2db7382-857f-4885-b350-6d461a370c1a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:10:03,455-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1a0000b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:10:03,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2ced2cbc] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1a0000b7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:10:03,468-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [34577400] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:10:03,469-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [34577400] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:15:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [548cd5d6] Autorecovering 2 hosts >2018-05-30 07:15:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [548cd5d6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:15:00,069-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:15:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,077-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78051c4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2bc5825d >2018-05-30 07:15:00,081-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [78051c4d] FINISH, SetVdsStatusVDSCommand, log id: 2bc5825d >2018-05-30 07:15:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,084-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [78051c4d] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:15:00,084-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [78051c4d] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:15:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:15:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:15:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [29143a6c] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5308d2ab >2018-05-30 07:15:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [29143a6c] FINISH, SetVdsStatusVDSCommand, log id: 5308d2ab >2018-05-30 07:15:00,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [29143a6c] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 61c36a36 >2018-05-30 07:15:00,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler8) [29143a6c] FINISH, SetHaMaintenanceModeVDSCommand, log id: 61c36a36 >2018-05-30 07:15:00,105-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:15:00,106-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [29143a6c] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:15:01,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2119db7e] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 15a65cd6 >2018-05-30 07:15:01,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [2119db7e] FINISH, GetHardwareInfoVDSCommand, log id: 15a65cd6 >2018-05-30 07:15:01,057-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [431422b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:15:01,066-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [773ca1d9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:15:01,071-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [67423b6d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:15:01,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [67423b6d] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21248cd1 >2018-05-30 07:15:01,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:15:01,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [67423b6d] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21248cd1 >2018-05-30 07:15:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:01,466-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:01,469-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:01,470-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:01,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:01,474-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler7) [67423b6d] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:01,477-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [67423b6d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:15:01,480-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [67423b6d] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 67423b6d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:15:01,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 75eafd7d >2018-05-30 07:15:01,486-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:15:01,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:15:01,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:15:01,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 75eafd7d >2018-05-30 07:15:01,584-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:15:01,597-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler7) [1675ab2f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:15:01,598-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1675ab2f] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5f98bbe7 >2018-05-30 07:15:01,600-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler7) [1675ab2f] FINISH, SetVdsStatusVDSCommand, log id: 5f98bbe7 >2018-05-30 07:15:01,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1675ab2f, Job ID: 8ba00a2a-f2e7-49d6-8ac2-577d4f3117c6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:15:01,636-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 773ca1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:15:01,638-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1675ab2f] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 773ca1d9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:15:01,648-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [19869c1d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:15:01,650-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [19869c1d] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:15:02,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [488493bb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 194bdb1d >2018-05-30 07:15:02,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [488493bb] FINISH, GetHardwareInfoVDSCommand, log id: 194bdb1d >2018-05-30 07:15:02,507-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [58e350c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:15:02,515-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5ab01986] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:15:02,521-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [311ece45] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:15:02,527-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [311ece45] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 58bc0b33 >2018-05-30 07:15:02,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:15:02,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [311ece45] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 58bc0b33 >2018-05-30 07:15:02,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:02,908-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:02,911-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:02,912-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:02,916-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:15:02,917-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [311ece45] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:15:02,919-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [311ece45] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:15:02,922-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [311ece45] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 311ece45, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:15:02,925-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7af9a9a3 >2018-05-30 07:15:02,925-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:15:03,023-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:15:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:15:03,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:15:03,024-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:15:03,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7af9a9a3 >2018-05-30 07:15:03,024-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:15:03,036-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [60444e4b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:15:03,037-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60444e4b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 457ed7c3 >2018-05-30 07:15:03,040-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [60444e4b] FINISH, SetVdsStatusVDSCommand, log id: 457ed7c3 >2018-05-30 07:15:03,045-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 60444e4b, Job ID: f44be41a-ff35-447b-a190-b02a59416d33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:15:03,049-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5ab01986, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:15:03,051-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [60444e4b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5ab01986, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:15:03,062-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [668a8984] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:15:03,063-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [668a8984] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [40cdf13b] Autorecovering 2 hosts >2018-05-30 07:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [40cdf13b] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:20:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,067-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7d4bcb88] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 698f0bb8 >2018-05-30 07:20:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7d4bcb88] FINISH, SetVdsStatusVDSCommand, log id: 698f0bb8 >2018-05-30 07:20:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7d4bcb88] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:20:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler3) [7d4bcb88] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:20:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7fbed063 >2018-05-30 07:20:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] FINISH, SetVdsStatusVDSCommand, log id: 7fbed063 >2018-05-30 07:20:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: a519257 >2018-05-30 07:20:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler3) [7b93f4dd] FINISH, SetHaMaintenanceModeVDSCommand, log id: a519257 >2018-05-30 07:20:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:20:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler3) [7b93f4dd] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:20:00,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [d01d73b] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 66d7ac9 >2018-05-30 07:20:00,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [d01d73b] FINISH, GetHardwareInfoVDSCommand, log id: 66d7ac9 >2018-05-30 07:20:00,796-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [31d33d3f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:20:00,801-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [2fef773d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:20:00,806-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d300865] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:20:00,818-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d300865] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 60d8e6b4 >2018-05-30 07:20:02,273-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [309d21cb] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 65f97ebb >2018-05-30 07:20:02,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler5) [309d21cb] FINISH, GetHardwareInfoVDSCommand, log id: 65f97ebb >2018-05-30 07:20:02,287-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler5) [420cc60a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:20:02,291-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler5) [74069] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:20:02,296-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2f93801c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:20:02,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2f93801c] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2d7bb419 >2018-05-30 07:20:02,733-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:20:02,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [2f93801c] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2d7bb419 >2018-05-30 07:20:02,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:02,738-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:02,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:02,742-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:02,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:02,746-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler5) [2f93801c] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:02,749-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler5) [2f93801c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:20:02,751-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2f93801c] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 2f93801c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:20:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 1041758 >2018-05-30 07:20:02,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:20:02,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:20:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:20:02,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:20:02,859-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:20:02,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-26) [] FINISH, ConnectStoragePoolVDSCommand, log id: 1041758 >2018-05-30 07:20:02,859-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-26) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:20:02,871-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler5) [4896be06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:20:02,872-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4896be06] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 48fa2f94 >2018-05-30 07:20:02,875-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [4896be06] FINISH, SetVdsStatusVDSCommand, log id: 48fa2f94 >2018-05-30 07:20:02,883-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 4896be06, Job ID: 96345814-ae7b-4bb0-a2d2-dc0bb366a7c3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:20:02,887-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 74069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:20:02,889-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4896be06] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 74069, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:20:02,900-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [41c80b5c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:20:02,901-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [41c80b5c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:20:03,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:20:03,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [4d300865] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 60d8e6b4 >2018-05-30 07:20:03,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:03,143-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:03,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:03,147-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:03,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:20:03,151-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler8) [4d300865] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:20:03,154-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [4d300865] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:20:03,157-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4d300865] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 4d300865, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:20:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7f133bac >2018-05-30 07:20:03,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-40) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:20:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:20:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:20:03,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7f133bac >2018-05-30 07:20:03,256-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-40) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:20:03,268-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [23c14c1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:20:03,269-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23c14c1b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33602198 >2018-05-30 07:20:03,271-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [23c14c1b] FINISH, SetVdsStatusVDSCommand, log id: 33602198 >2018-05-30 07:20:03,276-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 23c14c1b, Job ID: e7bf72de-c0a4-4705-9d35-13049f5ed7ea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:20:03,281-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2fef773d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:20:03,283-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [23c14c1b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2fef773d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:20:03,293-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [2d98cdae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:20:03,294-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2d98cdae] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [72981ac6] Autorecovering 2 hosts >2018-05-30 07:25:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [72981ac6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:25:00,062-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:25:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:25:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d839db6] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 1eddae4d >2018-05-30 07:25:00,073-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [2d839db6] FINISH, SetVdsStatusVDSCommand, log id: 1eddae4d >2018-05-30 07:25:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [2d839db6] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:25:00,076-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler10) [2d839db6] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:25:00,082-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,088-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7a846b94 >2018-05-30 07:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] FINISH, SetVdsStatusVDSCommand, log id: 7a846b94 >2018-05-30 07:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 621b164f >2018-05-30 07:25:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler10) [d5bafcf] FINISH, SetHaMaintenanceModeVDSCommand, log id: 621b164f >2018-05-30 07:25:00,096-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:25:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler10) [d5bafcf] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:25:00,632-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7c486e85] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 34fa973b >2018-05-30 07:25:00,634-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler4) [7c486e85] FINISH, GetHardwareInfoVDSCommand, log id: 34fa973b >2018-05-30 07:25:00,646-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler4) [59904137] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:25:00,655-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler4) [2f21b60e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:25:00,660-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [40da236a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:25:00,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [40da236a] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 505efdd4 >2018-05-30 07:25:00,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [712816b4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 56cf44ca >2018-05-30 07:25:00,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [712816b4] FINISH, GetHardwareInfoVDSCommand, log id: 56cf44ca >2018-05-30 07:25:00,951-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [442f75f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:25:00,956-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [2e157a02] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:25:00,961-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6607c63e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:25:00,967-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6607c63e] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 794bf8c3 >2018-05-30 07:25:01,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:25:01,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler4) [40da236a] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 505efdd4 >2018-05-30 07:25:01,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,059-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,064-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,067-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,068-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler4) [40da236a] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,070-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler4) [40da236a] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:25:01,073-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [40da236a] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 40da236a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:25:01,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 7415724e >2018-05-30 07:25:01,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:25:01,176-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-30) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:25:01,176-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:25:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:25:01,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:25:01,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-30) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7415724e >2018-05-30 07:25:01,177-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-30) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:25:01,189-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler4) [1b4f7735] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:25:01,190-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1b4f7735] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 69f35dce >2018-05-30 07:25:01,192-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler4) [1b4f7735] FINISH, SetVdsStatusVDSCommand, log id: 69f35dce >2018-05-30 07:25:01,225-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1b4f7735, Job ID: ca2bc444-b7d2-47b3-8490-285379a564be, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:25:01,229-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2f21b60e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:25:01,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1b4f7735] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2f21b60e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:25:01,241-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler4) [1b91bb68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:25:01,242-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [1b91bb68] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:25:01,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:25:01,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [6607c63e] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 794bf8c3 >2018-05-30 07:25:01,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,349-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,353-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,353-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:25:01,358-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [6607c63e] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:25:01,361-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [6607c63e] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:25:01,363-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6607c63e] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 6607c63e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:25:01,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: a1df0f2 >2018-05-30 07:25:01,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:25:01,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:25:01,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:25:01,465-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-36) [] FINISH, ConnectStoragePoolVDSCommand, log id: a1df0f2 >2018-05-30 07:25:01,465-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-36) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:25:01,477-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [695f7f4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:25:01,479-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [695f7f4d] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4c194a7 >2018-05-30 07:25:01,481-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [695f7f4d] FINISH, SetVdsStatusVDSCommand, log id: 4c194a7 >2018-05-30 07:25:01,486-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 695f7f4d, Job ID: ab0e1405-adef-4663-91b2-946cc51e8b0a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:25:01,490-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 2e157a02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:25:01,492-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [695f7f4d] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 2e157a02, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:25:01,502-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [7c6e0218] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:25:01,503-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7c6e0218] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [71cf5e96] Autorecovering 2 hosts >2018-05-30 07:30:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [71cf5e96] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:30:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c268f30] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: ad24737 >2018-05-30 07:30:00,075-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [c268f30] FINISH, SetVdsStatusVDSCommand, log id: ad24737 >2018-05-30 07:30:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,078-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [c268f30] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:30:00,078-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [c268f30] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:30:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:30:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,091-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [619f00af] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6ad19550 >2018-05-30 07:30:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [619f00af] FINISH, SetVdsStatusVDSCommand, log id: 6ad19550 >2018-05-30 07:30:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [619f00af] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 7328c72 >2018-05-30 07:30:00,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [619f00af] FINISH, SetHaMaintenanceModeVDSCommand, log id: 7328c72 >2018-05-30 07:30:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:30:00,099-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [619f00af] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:30:01,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6e832ca6] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 730827fb >2018-05-30 07:30:01,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6e832ca6] FINISH, GetHardwareInfoVDSCommand, log id: 730827fb >2018-05-30 07:30:01,865-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [75c2d8e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:30:01,870-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [4c905204] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:30:01,874-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5ff6c697] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:30:01,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5ff6c697] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2b52d70f >2018-05-30 07:30:02,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1872b398] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 1807730b >2018-05-30 07:30:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [1872b398] FINISH, GetHardwareInfoVDSCommand, log id: 1807730b >2018-05-30 07:30:02,127-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [724c944b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:30:02,131-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [408d35a1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:30:02,136-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7afbca16] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:30:02,142-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7afbca16] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 60c3de17 >2018-05-30 07:30:02,258-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:30:02,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [5ff6c697] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2b52d70f >2018-05-30 07:30:02,266-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:02,267-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:02,270-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:02,271-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:02,275-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:02,276-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler10) [5ff6c697] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:02,278-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [5ff6c697] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:30:02,281-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ff6c697] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 5ff6c697, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:30:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 73eee22b >2018-05-30 07:30:02,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:30:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:30:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:30:02,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 73eee22b >2018-05-30 07:30:02,383-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:30:02,396-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler10) [219cd652] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:30:02,397-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [219cd652] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 53e90279 >2018-05-30 07:30:02,399-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler10) [219cd652] FINISH, SetVdsStatusVDSCommand, log id: 53e90279 >2018-05-30 07:30:02,405-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 219cd652, Job ID: 36932046-fd1d-4390-9d89-c98fd6f4009a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:30:02,410-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 4c905204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:30:02,412-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [219cd652] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 4c905204, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:30:02,422-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [2b9dc66c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:30:02,423-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2b9dc66c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:30:03,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:30:03,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [7afbca16] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 60c3de17 >2018-05-30 07:30:03,511-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:03,512-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:03,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:03,517-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:03,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:30:03,521-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [7afbca16] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:30:03,524-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [7afbca16] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:30:03,527-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7afbca16] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 7afbca16, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:30:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: f5b383b >2018-05-30 07:30:03,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:30:03,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:30:03,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:30:03,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-35) [] FINISH, ConnectStoragePoolVDSCommand, log id: f5b383b >2018-05-30 07:30:03,629-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-35) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:30:03,642-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [46b79872] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:30:03,643-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46b79872] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 57d27d0f >2018-05-30 07:30:03,645-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [46b79872] FINISH, SetVdsStatusVDSCommand, log id: 57d27d0f >2018-05-30 07:30:03,651-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 46b79872, Job ID: e35ae683-4d3d-4911-9923-738428c1906f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:30:03,655-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 408d35a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:30:03,657-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [46b79872] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 408d35a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:30:03,668-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [35f11183] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:30:03,669-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [35f11183] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [539d36e6] Autorecovering 2 hosts >2018-05-30 07:35:00,008-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [539d36e6] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:35:00,061-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,070-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c91fb9] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 71feb9d4 >2018-05-30 07:35:00,072-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [73c91fb9] FINISH, SetVdsStatusVDSCommand, log id: 71feb9d4 >2018-05-30 07:35:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,075-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [73c91fb9] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:35:00,075-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [73c91fb9] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:35:00,081-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,087-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,088-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68e33015] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 342b39ea >2018-05-30 07:35:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [68e33015] FINISH, SetVdsStatusVDSCommand, log id: 342b39ea >2018-05-30 07:35:00,093-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [68e33015] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 4d54958 >2018-05-30 07:35:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler2) [68e33015] FINISH, SetHaMaintenanceModeVDSCommand, log id: 4d54958 >2018-05-30 07:35:00,095-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:35:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [68e33015] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:35:01,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b357218] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 2824c11d >2018-05-30 07:35:01,290-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b357218] FINISH, GetHardwareInfoVDSCommand, log id: 2824c11d >2018-05-30 07:35:01,303-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [6eb58fa5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:35:01,312-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [1baa5a3c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:35:01,317-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [57b8bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:35:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [57b8bf] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c49cc66 >2018-05-30 07:35:01,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:35:01,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [57b8bf] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 2c49cc66 >2018-05-30 07:35:01,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:01,708-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:01,711-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:01,712-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:01,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:01,716-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler2) [57b8bf] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:01,719-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [57b8bf] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:35:01,722-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57b8bf] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 57b8bf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:35:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 15f32612 >2018-05-30 07:35:01,728-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:35:01,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:35:01,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:35:01,830-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-46) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15f32612 >2018-05-30 07:35:01,830-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-46) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:35:01,844-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler2) [28a6480b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:35:01,845-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [28a6480b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 33e54af2 >2018-05-30 07:35:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [28a6480b] FINISH, SetVdsStatusVDSCommand, log id: 33e54af2 >2018-05-30 07:35:01,878-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 28a6480b, Job ID: c768bdeb-99d1-4759-8f57-a16bea8d9cea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:35:01,883-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1baa5a3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:35:01,884-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [28a6480b] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 1baa5a3c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:35:01,895-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [73a91267] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:35:01,896-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [73a91267] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com'(dad86010-4833-4906-85cb-5fd341611eed) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:35:03,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1c448d1a] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 5edad36f >2018-05-30 07:35:03,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1c448d1a] FINISH, GetHardwareInfoVDSCommand, log id: 5edad36f >2018-05-30 07:35:03,083-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [4e62389a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:35:03,092-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [17e95d8a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:35:03,096-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [507346d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:35:03,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [507346d5] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21f35cc9 >2018-05-30 07:35:03,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:35:03,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [507346d5] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 21f35cc9 >2018-05-30 07:35:03,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:03,497-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:03,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:03,502-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:03,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:35:03,506-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler3) [507346d5] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:35:03,509-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [507346d5] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:35:03,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [507346d5] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 507346d5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:35:03,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 4a92b76d >2018-05-30 07:35:03,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:35:03,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:35:03,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:35:03,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-7) [] FINISH, ConnectStoragePoolVDSCommand, log id: 4a92b76d >2018-05-30 07:35:03,614-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-7) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:35:03,627-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler3) [56178bad] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:35:03,628-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56178bad] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 719a973 >2018-05-30 07:35:03,630-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler3) [56178bad] FINISH, SetVdsStatusVDSCommand, log id: 719a973 >2018-05-30 07:35:03,661-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 56178bad, Job ID: 345e0509-69bd-48b1-9dcd-976dc18a7081, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:35:03,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 17e95d8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:35:03,668-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [56178bad] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 17e95d8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:35:03,679-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5a368bae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:35:03,680-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [5a368bae] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [d570fd8] Autorecovering 2 hosts >2018-05-30 07:40:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [d570fd8] Autorecovering hosts id: 17d87580-cef0-4aa1-beee-e0e773f805a1 , name : dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:40:00,063-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Before acquiring lock in order to prevent monitoring for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,070-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,071-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5575fd13] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3d727fee >2018-05-30 07:40:00,074-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5575fd13] FINISH, SetVdsStatusVDSCommand, log id: 3d727fee >2018-05-30 07:40:00,076-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5575fd13] Lock freed to object 'EngineLock:{exclusiveLocks='[17d87580-cef0-4aa1-beee-e0e773f805a1=VDS]', sharedLocks=''}' >2018-05-30 07:40:00,077-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [5575fd13] Autorecovering hosts id: dad86010-4833-4906-85cb-5fd341611eed , name : dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:40:00,083-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDSAction group MANIPULATE_HOST with role type ADMIN >2018-05-30 07:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Before acquiring lock in order to prevent monitoring for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,089-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock acquired, from now a monitoring of host will be skipped for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,090-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 201540e1 >2018-05-30 07:40:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] FINISH, SetVdsStatusVDSCommand, log id: 201540e1 >2018-05-30 07:40:00,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] START, SetHaMaintenanceModeVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetHaMaintenanceModeVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed'}), log id: 2462267a >2018-05-30 07:40:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (DefaultQuartzScheduler5) [58a97e5a] FINISH, SetHaMaintenanceModeVDSCommand, log id: 2462267a >2018-05-30 07:40:00,097-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Activate host finished. Lock released. Monitoring can run now for host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' from data-center 'Default' >2018-05-30 07:40:00,098-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [58a97e5a] Lock freed to object 'EngineLock:{exclusiveLocks='[dad86010-4833-4906-85cb-5fd341611eed=VDS]', sharedLocks=''}' >2018-05-30 07:40:01,293-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [18976593] START, GetHardwareInfoVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vds='Host[dell-per630-07.css.lab.eng.rdu2.redhat.com,17d87580-cef0-4aa1-beee-e0e773f805a1]'}), log id: 17cab53 >2018-05-30 07:40:01,295-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler9) [18976593] FINISH, GetHardwareInfoVDSCommand, log id: 17cab53 >2018-05-30 07:40:01,308-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler9) [3a038aa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:40:01,317-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler9) [5f30c767] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:40:01,322-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [cf0f20b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:40:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [cf0f20b] START, ConnectStorageServerVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 49affcb0 >2018-05-30 07:40:01,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:40:01,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler9) [cf0f20b] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 49affcb0 >2018-05-30 07:40:01,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:01,709-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:01,712-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:01,713-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:01,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:01,717-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler9) [cf0f20b] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:01,720-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler9) [cf0f20b] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:40:01,722-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [cf0f20b] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: cf0f20b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:40:01,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 2f121e2b >2018-05-30 07:40:01,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-07.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:40:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:40:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com >2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', vdsId='17d87580-cef0-4aa1-beee-e0e773f805a1', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:40:01,826-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2f121e2b >2018-05-30 07:40:01,826-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'dell-per630-07.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:40:01,838-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [5bfa1db8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:40:01,840-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bfa1db8] START, SetVdsStatusVDSCommand(HostName = dell-per630-07.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='17d87580-cef0-4aa1-beee-e0e773f805a1', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 74f705e4 >2018-05-30 07:40:01,842-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [5bfa1db8] FINISH, SetVdsStatusVDSCommand, log id: 74f705e4 >2018-05-30 07:40:01,873-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5bfa1db8, Job ID: 730d48f5-203c-4455-90f5-16424860ac74, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-07.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:40:01,877-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 5f30c767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-07.css.lab.eng.rdu2.redhat.com. >2018-05-30 07:40:01,879-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5bfa1db8] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Correlation ID: 5f30c767, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-07.css.lab.eng.rdu2.redhat.com to Storage Pool Default >2018-05-30 07:40:01,890-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler9) [13f5eb0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 17d87580-cef0-4aa1-beee-e0e773f805a1 Type: VDS >2018-05-30 07:40:01,891-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [13f5eb0c] Host 'dell-per630-07.css.lab.eng.rdu2.redhat.com'(17d87580-cef0-4aa1-beee-e0e773f805a1) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. >2018-05-30 07:40:02,537-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7f111ec4] START, GetHardwareInfoVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vds='Host[dell-per630-06.css.lab.eng.rdu2.redhat.com,dad86010-4833-4906-85cb-5fd341611eed]'}), log id: 480f7c2c >2018-05-30 07:40:02,539-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [7f111ec4] FINISH, GetHardwareInfoVDSCommand, log id: 480f7c2c >2018-05-30 07:40:02,551-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [7f5e62e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:40:02,560-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [1948a1c7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:40:02,565-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [776dea78] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5b05a987-0149-0267-0212-0000000001df Type: StoragePool >2018-05-30 07:40:02,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [776dea78] START, ConnectStorageServerVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03', connection='192.168.50.21:/vmstore', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='84ab6a6d-cbbc-4d66-be44-443691c12a81', connection='192.168.50.21:/engine', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b09bb99a-3efc-4645-a73f-e8d81dce8e2e', connection='192.168.50.21:/data', iqn='null', vfsType='glusterfs', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1974cc84 >2018-05-30 07:40:02,936-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: VDS_STORAGES_CONNECTION_FAILED(188), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to the Storage Domains vmstore,data,hosted_storage. >2018-05-30 07:40:02,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [776dea78] FINISH, ConnectStorageServerVDSCommand, return: {c56e136b-a8f3-4abd-86c5-9e2d8cdb5c03=477, b09bb99a-3efc-4645-a73f-e8d81dce8e2e=477, 84ab6a6d-cbbc-4d66-be44-443691c12a81=477}, log id: 1974cc84 >2018-05-30 07:40:02,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/vmstore returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:02,940-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/vmstore' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:02,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/data returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:02,945-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/data' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:02,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: STORAGE_DOMAIN_ERROR(996), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: The error message for connection 192.168.50.21:/engine returned by VDSM was: Problem while trying to mount target >2018-05-30 07:40:02,949-04 ERROR [org.ovirt.engine.core.bll.storage.connection.BaseFsStorageHelper] (DefaultQuartzScheduler1) [776dea78] The connection with details '192.168.50.21:/engine' failed because of error code '477' and error message is: problem while trying to mount target >2018-05-30 07:40:02,952-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [776dea78] Host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' storage connection was failed >2018-05-30 07:40:02,954-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [776dea78] EVENT_ID: CONNECT_STORAGE_SERVERS_FAILED(994), Correlation ID: 776dea78, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to connect Host dell-per630-06.css.lab.eng.rdu2.redhat.com to Storage Servers >2018-05-30 07:40:02,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'}), log id: 74a0cbd6 >2018-05-30 07:40:02,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {2afb6de5-5377-4aa6-853a-4e4023ce0212=active, dc1e563f-7a35-44cf-bffb-b8623a5b0be5=active, 8b8a4f48-ad72-4a7f-a7a3-5353dfe6d147=active} >2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-6) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM dell-per630-06.css.lab.eng.rdu2.redhat.com command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:40:03,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5']]' >2018-05-30 07:40:03,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com >2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Command 'ConnectStoragePoolVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', vdsId='dad86010-4833-4906-85cb-5fd341611eed', storagePoolId='5b05a987-0149-0267-0212-0000000001df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=5b05a987-0149-0267-0212-0000000001df, msdUUID=dc1e563f-7a35-44cf-bffb-b8623a5b0be5' >2018-05-30 07:40:03,053-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 74a0cbd6 >2018-05-30 07:40:03,053-04 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-6) [] Could not connect host 'dell-per630-06.css.lab.eng.rdu2.redhat.com' to pool 'Default': null >2018-05-30 07:40:03,065-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler1) [1da0861b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: dad86010-4833-4906-85cb-5fd341611eed Type: VDS >2018-05-30 07:40:03,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1da0861b] START, SetVdsStatusVDSCommand(HostName = dell-per630-06.css.lab.eng.rdu2.redhat.com, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='dad86010-4833-4906-85cb-5fd341611eed', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: f5d2379 >2018-05-30 07:40:03,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler1) [1da0861b] FINISH, SetVdsStatusVDSCommand, log id: f5d2379 >2018-05-30 07:40:03,098-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1da0861b] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 1da0861b, Job ID: 971be869-4806-4b01-b3da-c43be59d8c8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host dell-per630-06.css.lab.eng.rdu2.redhat.com cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational. >2018-05-30 07:40:03,102-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [1da0861b] EVENT_ID: VDS_ALERT_FENCE_IS_NOT_CONFIGURED(9,000), Correlation ID: 1948a1c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host dell-per630-06.css.lab.eng.rdu2.redhat.com. >